[Volute] r4045 - trunk/projects/dm/vo-dml/mapping/MAST VODML Parser

Volute commit messages volutecommits at g-vo.org
Wed May 10 15:46:21 CEST 2017


Author: tdonaldson
Date: Wed May 10 15:46:21 2017
New Revision: 4045

Log:
Update MAST Parser documentation.

Added:
   trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/link_32x32.png   (contents, props changed)
Modified:
   trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/VODML Parser.html
   trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/runAll.sh

Modified: trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/VODML Parser.html
==============================================================================
--- trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/VODML Parser.html	Wed May 10 13:39:40 2017	(r4044)
+++ trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/VODML Parser.html	Wed May 10 15:46:21 2017	(r4045)
@@ -1,17 +1,45 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
 <html><head>
 
+
   
   <meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>VODML Parser</title></head><body>
 <h1>VODML Parser Service<br>
 </h1>
 
 <h2>Summary</h2>
-
+<h4>Service</h4>
 This web service transforms a VOTABLE into a variety of formats. 
-The input table can contain the proposed VODML mapping syntax or the
-alternate mapping syntax (see votableAlt option format option below),
-but not both.  <br>
+The input table can contain the proposed VODML mapping syntax (<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/xsd/ext/VOTable-1.4_vodml.xsd" target="_blank">VOTable-1.4_vodml.xsd</a>).<br>
+<h4>Client</h4>
+In addition to the web service, the MAST Portal client can display the
+content of VODML annotated VOTABLEs.  Use the link below to active
+that version of the MAST Portal, then load the VOTABLE via URL in the
+search box or via "Upload Target List" (really means upload any table).<br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Clients/Mast/Portal.html" target="_blank">https://masttest.stsci.edu/vodml/Mashup/Clients/Mast/Portal.html</a><br>
+After loading a table via URL, you can grab a shortcut link to the
+Portal with that table loaded using the "Copy Link for this Search"
+button.  <img style="width: 32px; height: 32px;" alt="Link" src="link_32x32.png"><br>
+<br>
+<h2>Notes / Todo<br>
+</h2>
+
+
+<ul>
+<li><span style="font-weight: bold;"></span>The parser doesn't currently parse FITS files, so 100 rows of fake data are generated for VOTABLEs that use FITS.  (Once the parser can recognize FITS, it still won't be able to find "file:" references to those FITS files.</li>
+  <li>The template filling and reference following happens only in the
+client now.  Maybe a future extension of the parser can do that
+server-side as well.</li>
+  <li>The client only displays the first TABLE in the VOTABLE.  This needs to be enhanced.</li>
+  <li>The client doesn't dereference and REMOTEREFERENCE or FOREIGNKEY references.</li>
+  <li>The client should demonstrate that it "knows" certain models (e.g., sample, CAOM).</li>
+  <ul>
+    <li>This could/should include validating that the model matches the VODML annotations.</li>
+  </ul>
+  <li><br>
+  </li>
+</ul>
+
 
 <h2>Arguments</h2>
 
@@ -19,8 +47,7 @@
 
   <li><span style="font-weight: bold;">format</span> - The desired
 output format.  Available choices (not case sensitive) are
-below.  The first 3 are the ones that may be useful for VODML
-discussions:</li>
+below:</li>
   <br>
   <ul>
     <li><span style="font-weight: bold;">json</span> - A JSON
@@ -29,14 +56,7 @@
     </li>
     <li><span style="font-weight: bold;">jsonlite</span> - A slightly
 shorter serialization of all the VODML instances.  The only real
-difference is in the leaf nodes for primitive data values.</li>
-    <li><span style="font-weight: bold;">vodmli</span> - A VODML-I formatted xml, hopefully obeying this schema:  <a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/xsd/vo-dml-instance.xsd" target="_blank">vo-dml-instance.xsd</a><br>
-    </li>
-
-    <li><span style="font-weight: bold;">votableAlt</span> - A VOTABLE
-equivalent to the input VOTABLE but with the VODML instances serialized
-using an alternative mapping.</li>
-    <li><span style="font-weight: bold;">extjs</span> - A complex JSON
+difference is in the leaf nodes for primitive data values.</li><li><span style="font-weight: bold;">extjs</span> - A complex JSON
 serialization intended for use by the MAST Discivery Portal.  This
 format contains all the data from the original table along with some
 client-specific information such as histograms of the data
@@ -82,12 +102,10 @@
   </ul>
   <ul>
     <li>This example URL, which can be used with <span style="font-style: italic;">curl</span> or in a browser, parses
-test4.votable.xml from volute, and outputs the resulting VODML
+test5.votable-1.4.xml from volute, and outputs the resulting VODML
 instances in <span style="font-weight: bold;">json</span> format:</li>
   </ul>
-</ul>
-
-<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test4.votable.xml" target="_blank">https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test4.votable.xml</a><br>
+</ul><a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test5.votable-1.4.xml" target="_blank">https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test5.votable-1.4.xml</a><br>
 
 <ul>
 
@@ -101,7 +119,7 @@
   </ul>
 </ul>
 
-<pre>curl \<br>  -F "format=json" \<br>  -F "votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test4.votable.xml" \<br>  https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodmlForm<br><br></pre>
+<pre>curl \<br>  -F "format=json" \<br>  -F "votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test5.votable-1.4.xml" \<br>  https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodmlForm<br><br></pre>
 
 <ul>
 
@@ -111,88 +129,27 @@
   </ul>
 </ul>
 
-<pre>curl \<br>  -F "format=json" \<br>  -F "votable=@/Users/tom/myfiles/test4.votable.xml" \<br> https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodmlForm<br></pre>
+<pre>curl \<br>  -F "format=json" \<br>  -F "votable=@/Users/tom/myfiles/test5.votable-1.4.xml" \<br> https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodmlForm<br></pre>
 <h2><br>
 </h2>
-<h2>Sample Output</h2>
-The parser was run on some of the volute test files with output stored in subdirectories by format.<br>
-<h3>Input Files</h3>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test1.votable.xml" target="_blank">test1.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test2.votable.xml" target="_blank">test2.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test3.votable.xml" target="_blank">test3.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test4.votable.xml" target="_blank">test4.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/Association/Association.votable.xml" target="_blank">Association.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/Composition/Composition.votable.xml" target="_blank">Composition.votable.xml</a><br>
-<a href="https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/EnumLiteral/EnumLiteral.votable.xml" target="_blank">EnumLiteral.votable.xml</a><br>
-<br>
-This script runs all the above input files through the parser,
-generating all the output format into the following directories
-(relative to where the script was run).  The links in the
-directories below connect to the committted versions of those output
-directories in SVN.<br>
-<a href="runAll.sh" target="_blank">runAll.sh</a><br>
-<h3>Output Directories</h3>
-<a href="outJson" target="_blank">outJson</a><br>
-<br>
-<a href="outJsonLite" target="_blank">outJsonLite</a><br>
-<br>
-<a href="outMast" target="_blank">outMast</a> - A MAST-specific format that contains a variety of additional data to assist the MAST client<br>
-
-<br>
+<h2>Examples<br>
+</h2>
+<h4>Create JSON format for some annotated VOTABLEs in volute</h4>
 
-<a href="outVodmli" target="_blank">outVodmli</a> - Vodmli format output<br>
-<br>
-<a href="outVotable" target="_blank">outVotable</a> - An attempt to recreate a VOTABLE equivalent to the input one.  Possibly useful in round trip regression tests.<br>
-<br>
-<a href="outVotableAlt" target="_blank">outVotableAlt</a> - These files are in the alternate VODML syntax, and can also be parsed by this same parser.  The schema is <a href="../../xsd/ext/VOTable-1.3_vodmlAlt.xsd" target="_blank">VOTable-1.3_vodmlAlt.xsd</a><br>
-<br>
-<br>
-<br>
-<h2>Output Formats</h2>
-The output formats are based on the likely oversimplified notion that
-the VODML entities represented in the VOTABLE fall into 3 categories:<br>
-<ul>
-  <li>Models</li>
-  <li>Global instances</li>
-  <li>Template instances which are defined within a TABLE, and imply one concrete instance per table row.</li>
-</ul>
-In the JSON formats, the Global instances are stored in 2 structures:<br>
-<ul>
-  <li><span style="font-weight: bold;">globalsById</span> - Instances with id's can be looked up in this dictionary where the keys are the id's.</li>
-  <li><span style="font-weight: bold;">nonIdGlobals</span> - An array of all the global instances that did not have id's.</li>
-</ul>
-In the JSON formats, there is an array of tables.  Each of those
-has an array of rows, which in turn contain an array of instances.<br>
-<br>
-In the alternative mapping VOTABLEs, the templates remain templates, with the table data staying in normal VOTABLE format.<br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test5.votable-1.4.xml" target="_blank">test5.votable-1.4.xml</a><br>
 <br>
-<h3>VODML-I Notes</h3>
-These are some items that this parser has difficulty with, usually
-because the parser does not read the model itself, and the VOTABLE does
-not contain the desired information:<br>
-<br>
-    •    I know the vodmlRef (from the
-ROLE), but I don’t know the “name”.  This value if often the tail
-end of the vodmlRef, but doesn’t have to be.<br>
-    •    For each GROUP I see, I don’t
-have any way to know whether to write out an object (object type) or a
-dataObject (value type)<br>
-    ⁃    My transientIDs will be
-internally consistent, but may not match those of others because these
-only count objects and I may have more or less of those than others.<br>
-    ⁃    <br>
-    •    I don’t know whether something belongs in a collection or not.<br>
-    •    I don’t know whether a primitive
-is representing a *Quantity.  I can guess that it is if it has a
-unit attribute.  If it does represent a *Quantity, then which type
-of quantity is it (see next issue)?<br>
-    •    Given a primitive (from a PARAM
-or FIELD), I may only know the primitive VOTABLE type.  How do I
-know the VODML type.<br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/tesselation/%20votable.annotatedRef.xml" target="_blank">votable.annotatedRef.xml</a><br>
 <br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/CubeDM-1.0/examples/chandra_2Dsky_image_annotated.vot" target="_blank">chandra_2Dsky_image_annotated.vot</a><br>
 <br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/CubeDM-1.0/examples/chandra_2Dsky_image_annotated_byref.vot" target="_blank">chandra_2Dsky_image_annotated_byref.vot</a><br>
 <br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Mashup.asmx/parseVodml?format=json&votable=https://volute.g-vo.org/svn/trunk/projects/dm/CubeDM-1.0/examples/chandra_events_annotated.vot" target="_blank">chandra_events_annotated.vot</a><br>
 <br>
+<h4>Run MAST portal on some annotated VOTABLEs in volute</h4>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Clients/Mast/Portal.html?searchQuery=%7B%22service%22%3A%22VOTABLE%22%2C%22inputText%22%3A%22https%3A%2F%2Ftwmastiisv4.stsci.edu%2Fvodml%2Fmashup%2Ftemp%2Ftest5.votable-1.4.xml%22%2C%22paramsService%22%3A%22Vo.Generic.Table%22%2C%22title%22%3A%22test5.votable-1.4.xml%22%2C%22columns%22%3A%22*%22%7D" target="_blank">test5.votable-1.4.xml</a><br>
+<a href="https://masttest.stsci.edu/vodml/Mashup/Clients/Mast/Portal.html?searchQuery=%7B%22service%22%3A%22VOTABLE%22%2C%22inputText%22%3A%22https%3A%2F%2Fvolute.g-vo.org%2Fsvn%2Ftrunk%2Fprojects%2Fdm%2FCubeDM-1.0%2Fexamples%2Fchandra_2Dsky_image_annotated.vot%22%2C%22paramsService%22%3A%22Vo.Generic.Table%22%2C%22title%22%3A%22https%3A%2F%2Fvolute.g-vo.org%2Fsvn%2Ftrunk%2Fprojects%2Fdm%2FCubeDM-1.0%2Fexamples%2Fchandra_2Dsky_image_annotated.vot%22%2C%22columns%22%3A%22*%22%7D" target="_blank"><br>
+chandra_2Dsky_image_annotated.vot</a><br>
 <br>
 <br>
 <br>

Added: trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/link_32x32.png
==============================================================================
Binary file. No diff available.

Modified: trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/runAll.sh
==============================================================================
--- trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/runAll.sh	Wed May 10 13:39:40 2017	(r4044)
+++ trunk/projects/dm/vo-dml/mapping/MAST VODML Parser/runAll.sh	Wed May 10 15:46:21 2017	(r4045)
@@ -19,23 +19,9 @@
 mkdir -p outVotable
 rm outVotable/*
 
-mkdir -p outVotableAlt
-rm outVotableAlt/*
-
-mkdir -p outVodmli
-rm outVodmli/*
-
-# test1.votable.xml
-
 for file in \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test1.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test2.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test3.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test4.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/Association/Association.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/Composition/Composition.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/EnumLiteral/EnumLiteral.votable.xml \
- https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/models/sample/EnumLiteral/EnumLiteral.votable.xml 
+ "@/Users/donald/dev/SMALL-VOLUTE/root/projects/dm/vo-dml/mapping/test5.votable-1.4.xml"
+# https://volute.g-vo.org/svn/trunk/projects/dm/vo-dml/mapping/test5.votable-1.4.xml
 do
 base=$(basename $file)
 
@@ -52,22 +38,12 @@
 curl \
   -F "format=extjs" \
   -F "votable=$file" \
-  $server/Mashup.asmx/parseVodmlForm > outMast/$base.mast.xml
+  $server/Mashup.asmx/parseVodmlForm > outMast/$base.mast.json
 
 curl \
   -F "format=votable" \
   -F "votable=$file" \
   $server/Mashup.asmx/parseVodmlForm > outVotable/$base.votable.xml
 
-curl \
-  -F "format=votableAlt" \
-  -F "votable=$file" \
-  $server/Mashup.asmx/parseVodmlForm > outVotableAlt/$base.votableAlt.xml
-
-curl \
-  -F "format=vodmli" \
-  -F "votable=$file" \
-  $server/Mashup.asmx/parseVodmlForm > outVodmli/$base.vo-dml.xml
-
 done
 


More information about the Volutecommits mailing list