# [Volute] r3563 - trunk/projects/registry/RegistryInterface

Volute commit messages volutecommits at g-vo.org
Wed Oct 26 18:05:50 CEST 2016

Author: msdemlei
Date: Thu Sep 22 13:47:22 2016
New Revision: 3563

Log:
RI 1.1: (whitespace only) reflowing overly long lines.

Modified:
trunk/projects/registry/RegistryInterface/RegistryInterface.tex

Modified: trunk/projects/registry/RegistryInterface/RegistryInterface.tex
==============================================================================
--- trunk/projects/registry/RegistryInterface/RegistryInterface.tex	Thu Sep 22 13:28:08 2016	(r3562)
+++ trunk/projects/registry/RegistryInterface/RegistryInterface.tex	Thu Sep 22 13:47:22 2016	(r3563)
@@ -46,11 +46,18 @@
\begin{document}

\begin{abstract}
-The VO Registry provides a mechanism with which VO applications can discover
-and select resources that are relevant for a particular scientific problem. This specification defines the operation of this system. It is based on a general,
-distributed model composed of searchable and publishing registries, as introduced at the beginning of this document. The main body of the specification has two components: (a) an interface for harvesting publishing registries, which builds upon the Open Archives Initiative Protocol for Metadata Harvesting.  (b) A VOResource extension for registering registry services and description of a central list of said IVOA registry services.
-Finally, this specification briefly discusses client interfaces to
-the Registry as provided by searchable registries.
+The VO Registry provides a mechanism with which VO applications can
+discover and select resources that are relevant for a particular
+scientific problem. This specification defines the operation of this
+system. It is based on a general, distributed model composed of
+searchable and publishing registries, as introduced at the beginning of
+this document. The main body of the specification has two components:
+(a) an interface for harvesting publishing registries, which builds upon
+the Open Archives Initiative Protocol for Metadata Harvesting.  (b) A
+VOResource extension for registering registry services and description
+of a central list of said IVOA registry services.  Finally, this
+specification briefly discusses client interfaces to the Registry as
+provided by searchable registries.
\end{abstract}

@@ -513,26 +520,52 @@

\label{sect:timegranularity}

-Datestamps in the OAI-PMH 2.0 standard are encoded using ISO8601 and expressed in UTC, with the UTC designator "z" appended to seconds-based granularity where supplied, i.e. \texttt{YYYY-MM-DDThh:mm:ssZ}. In general OAI-PMH registries, granularity at seconds scale is optional. Harvestable IVOA registries MUST report datestamps at the granularity of seconds and accept "from" and "until" arguments in the same format. This simplifies the incremental harvesting process in the multi-registry IVOA environment.
+Datestamps in the OAI-PMH 2.0 standard are encoded using ISO8601 and
+expressed in UTC, with the UTC designator "z" appended to seconds-based
+granularity where supplied, i.e. \texttt{YYYY-MM-DDThh:mm:ssZ}. In
+general OAI-PMH registries, granularity at seconds scale is optional.
+Harvestable IVOA registries MUST report datestamps at the granularity of
+seconds and accept "from" and "until" arguments in the same format. This
+simplifies the incremental harvesting process in the multi-registry IVOA
+environment.

\section{Registering Registries}

\label{regreg}

-Harvesting registries must able to locate remote registry resources relevant to them,
-and both harvesting registries and clients need access to metadata for the registry service itself. We address both of these issues by providing a schema for describing registries themselves, and a repository for indexing them.
-
-The resource specification for registries defines a VOResource extension schema called
-VORegistry, which describes provenance of the registry itself and its support for various interfaces described in this document or elsewhere. These VORegistry resources may themselves be stored as records in registries; each publishing registry MUST contain a self-descriptive VORegistry resource. VORegistry resources also include a list of naming authorities, where each represents a registry publisher's claim of ownership of an authority identifier. From each identifier, further IVOA identifiers for individual service or other records belonging under that publishing umbrella may be created. A publishing registry is said to exclusively manage a naming authority on behalf of the owning publisher; this means that within the IVOA registry network, only that specific registry may publish records having identifiers which begin with that authority identifier.
-
-The XML namespace URI of this schema is \nolinkurl{http://www.ivoa.net/xml/VORegistry/v1.0}.
-It has been chosen to allow it to be resolved as a URL to the XML Schema document, which is also given in appendix~\ref{app:vgschema}. The recommended prefix for this namespace is \xmlel{vg:}.
+Harvesting registries must able to locate remote registry resources
+relevant to them, and both harvesting registries and clients need access
+to metadata for the registry service itself. We address both of these
+issues by providing a schema for describing registries themselves, and a
+repository for indexing them.
+
+The resource specification for registries defines a VOResource extension
+schema called VORegistry, which describes provenance of the registry
+itself and its support for various interfaces described in this document
+or elsewhere. These VORegistry resources may themselves be stored as
+records in registries; each publishing registry MUST contain a
+self-descriptive VORegistry resource. VORegistry resources also include
+a list of naming authorities, where each represents a registry
+publisher's claim of ownership of an authority identifier. From each
+identifier, further IVOA identifiers for individual service or other
+records belonging under that publishing umbrella may be created. A
+publishing registry is said to exclusively manage a naming authority on
+behalf of the owning publisher; this means that within the IVOA registry
+network, only that specific registry may publish records having
+identifiers which begin with that authority identifier.
+
+The XML namespace URI of this schema is
+to allow it to be resolved as a URL to the XML Schema document, which is
+also given in appendix~\ref{app:vgschema}. The recommended prefix for
+this namespace is \xmlel{vg:}.

The schema has not been changed from the one used in version 1.0,
although the standard contents have somewhat changed.  The rationale for
-keeping the schema is that some schema features being no longer relevant has no
-detrimental consequences for Registry operations, whereas breaking clients with
-a change of the schema and XML namespace URI might have.
+keeping the schema is that some schema features being no longer relevant
+has no detrimental consequences for Registry operations, whereas
+breaking clients with a change of the schema and XML namespace URI might
+have.

\subsection{The Authority Resource Extension and the Publishing Process}
@@ -686,7 +719,9 @@
within its \texttt{ivo\_managed} set.

As of version 1.1 of this specification, VO registry records must provide
-the three mandatory VOSI capabilities: availability, a listing of service capabilities, and a listing of tables if relevant. \citep{std:VOSI}.
+the three mandatory VOSI capabilities: availability, a listing of
+service capabilities, and a listing of tables if relevant.
+\citep{std:VOSI}.

\subsection{The Search Capability}
@@ -734,27 +769,96 @@

\label{sect:rofr}

-To facilitate discovery and automated harvesting of registries containing VOResource records, a registry serving as a master list of IVOA registries exists as part of the IVOA web infrastructure, hosted at \nolinkurl{http://rofr.ivoa.net}. It is referred to as the Registry of Registries, or RofR (pronounced "rover"). As the RofR is itself a registry, an OAI-PMH interface is provided which conforms to this document. The OAI-PMH interface is always available at \nolinkurl{http://rofr.ivoa.net/oai}.
-
-The Registry of Registries includes the VOResource records directly representing each currently active registry of IVOA resources, be they fully searchable or publishing registries providing only an OAI-PMH harvesting interface. These resources are of type \xmlel{vg:Registry} as defined in section \ref{sect:resext}.
-
-Once a registry provider has deployed a new publishing registry, they must enroll it the RofR for searchable registries (and therefore registry search clients) to be able to find their records. The RofR provides a dedicated web-based interface for this purpose accessible from http://\nolinkurl{http://rofr.ivoa.net}.  The RofR includes a validator package, which thoroughly checks the new registry, including schema validation for the OAI interface itself and all listed resources. The registration process will only accept registries that validate successfully.  Local updates within a publishing registry post-inclusion in the RofR are not necessarily automatically validated by the RofR software later: the validator tool can, and indeed should, be used independently of the first admission process by the registry providers to periodically make sure their registries are still compliant with the relevant IVOA standards.
-
-The Registry of Registries also contains the canonical VOResource descriptions of the most recent versions of VOResource standards and extensions themselves, which are of type \xmlel{vstd:Standard}.
+To facilitate discovery and automated harvesting of registries
+containing VOResource records, a registry serving as a master list of
+IVOA registries exists as part of the IVOA web infrastructure, hosted at
+\nolinkurl{http://rofr.ivoa.net}. It is referred to as the Registry of
+Registries, or RofR (pronounced "rover"). As the RofR is itself a
+registry, an OAI-PMH interface is provided which conforms to this
+document. The OAI-PMH interface is always available at
+
+The Registry of Registries includes the VOResource records directly
+representing each currently active registry of IVOA resources, be they
+fully searchable or publishing registries providing only an OAI-PMH
+harvesting interface. These resources are of type \xmlel{vg:Registry} as
+defined in section \ref{sect:resext}.
+
+Once a registry provider has deployed a new publishing registry, they
+must enroll it the RofR for searchable registries (and therefore
+registry search clients) to be able to find their records. The RofR
+provides a dedicated web-based interface for this purpose accessible
+from http://\nolinkurl{http://rofr.ivoa.net}.  The RofR includes a
+validator package, which thoroughly checks the new registry, including
+schema validation for the OAI interface itself and all listed resources.
+The registration process will only accept registries that validate
+successfully.  Local updates within a publishing registry post-inclusion
+in the RofR are not necessarily automatically validated by the RofR
+software later: the validator tool can, and indeed should, be used
+independently of the first admission process by the registry providers
+to periodically make sure their registries are still compliant with the
+relevant IVOA standards.
+
+The Registry of Registries also contains the canonical VOResource
+extensions themselves, which are of type \xmlel{vstd:Standard}.

\subsection{Harvesting the Registry of Registries}

\label{sect:harvestrofr}

-Given the Registry of Registries contains records for all other currently active and validated IVOA registries, a client wishing to harvest the contents of all registries should begin at the RofR. Fully searchable registries wishing to include records from the other IVOA registries count among these potential clients. To harvest the entire contents of IVOA registries, it is recommended to first harvest the Registry of Registries via its OAI-PMH interface.
-
-This first step is done by making a call to the RofR's OAI-PMH interface with the\textbf{ListRecords} operation, with the \textbf{set} argument set to \textbf{ivo\_publishers}. This will return the registry records (i.e. resources with xsi:type='vg:Registry') for the registries that successfully registered themselves as described in \ref{sect:rofr}.
-
-The next step in harvesting the entire distributed IVOA registry contents is to iterate over the \xmlel{accessURL} of each \xmlel{vg:Registry} record's \xmlel{vr:capability} of type \xmlel{vg:Harvest}, and use the url for each of those OAI-PMH interfaces to harvest the individual registries. This filtering of RofR contents can be done by adding the \texttt{set} parameter to an OAI query to the RofR: registries in the RofR comprise the supported set \texttt{ivo\_publishers}. Then when harvesting each registry in turn, to avoid harvesting duplicate records from the fully searchable registries, it is recommended to add the \texttt{set} parameter to that OAI query: records specifically published by a registry which also has a search interface comprise that registry's supported set \texttt{ivo\_managed}.
-
-The very first time the harvester executes the \textbf{ListRecords} operation on the RofR or any listed registry, the \textbf{from} argument should be not used so that all known publishing registries are returned, as well as all known resources within each discovered registry. If the harvesting client wishes to use the OAI interface for incremental updates, it can cache at least a mapping of the registry identifiers to their respective harvesting endpoints along with a timestamp for when this operation was last successfully carried out on each. Then, at the start of subsequent harvesting updates, the harvester can provide the cached date using the \textbf{from} argument to receive only new and updated records, and update the cached timestamp upon success. It is suggested that harvesting clients perform full updates without the \textbf{from} parameter on an occasional basis.
-
-For example, to get a listing of registries in the IVOA ecosystem, one would first query \nolinkurl{http://rofr.ivoa.net/oai?verb=ListRecords\&metadataPrefix=ivo\_vor\&set=ivo_publishers}. Then, for each returned resource, the \xmlel{accessURL} under a \xmlel{Capability} with \xmlel{xsi:type=vg:Harvest}, that URL could be called as such: \nolinkurl{http://accessURLValue?verb=ListRecords\&metadataPrefix=ivo\_vor} or \nolinkurl{http://accessURLValue?verb=ListRecords\&metadataPrefix=ivo_vor\&from=YYYY-MM-DD} for return visits, with YYYY-MM-DD representing the last successful query to that accessURL.
+Given the Registry of Registries contains records for all other
+currently active and validated IVOA registries, a client wishing to
+harvest the contents of all registries should begin at the RofR. Fully
+searchable registries wishing to include records from the other IVOA
+registries count among these potential clients. To harvest the entire
+contents of IVOA registries, it is recommended to first harvest the
+Registry of Registries via its OAI-PMH interface.
+
+This first step is done by making a call to the RofR's OAI-PMH interface
+with the\textbf{ListRecords} operation, with the \textbf{set} argument
+set to \textbf{ivo\_publishers}. This will return the registry records
+(i.e. resources with xsi:type='vg:Registry') for the registries that
+successfully registered themselves as described in \ref{sect:rofr}.
+
+The next step in harvesting the entire distributed IVOA registry
+contents is to iterate over the \xmlel{accessURL} of each
+\xmlel{vg:Registry} record's \xmlel{vr:capability} of type
+\xmlel{vg:Harvest}, and use the url for each of those OAI-PMH interfaces
+to harvest the individual registries. This filtering of RofR contents
+can be done by adding the \texttt{set} parameter to an OAI query to the
+RofR: registries in the RofR comprise the supported set
+\texttt{ivo\_publishers}. Then when harvesting each registry in turn, to
+avoid harvesting duplicate records from the fully searchable registries,
+it is recommended to add the \texttt{set} parameter to that OAI query:
+interface comprise that registry's supported set \texttt{ivo\_managed}.
+
+The very first time the harvester executes the \textbf{ListRecords}
+operation on the RofR or any listed registry, the \textbf{from} argument
+should be not used so that all known publishing registries are returned,
+as well as all known resources within each discovered registry. If the
+harvesting client wishes to use the OAI interface for incremental
+updates, it can cache at least a mapping of the registry identifiers to
+their respective harvesting endpoints along with a timestamp for when
+this operation was last successfully carried out on each. Then, at the
+start of subsequent harvesting updates, the harvester can provide the
+cached date using the \textbf{from} argument to receive only new and
+updated records, and update the cached timestamp upon success. It is
+suggested that harvesting clients perform full updates without the
+\textbf{from} parameter on an occasional basis.
+
+For example, to get a listing of registries in the IVOA ecosystem, one
+would first query
+Then, for each returned resource, the \xmlel{accessURL} under a
+\xmlel{Capability} with \xmlel{xsi:type=vg:Harvest}, that URL could be
+called as such:
+or
+for return visits, with YYYY-MM-DD representing the last successful
+query to that accessURL.

\section{Searching the Registry}

@@ -771,15 +875,28 @@
has progressed to become an IVOA recommendation is RegTAP
\citep{std:RegTAP}, an interface based on a relational representation of
major parts of VOResource and the VO's TAP protocol \citep{std:TAP}.
-RegTAP services have been made available from several registry providers listed in the Registry of Registries.
+RegTAP services have been made available from several registry providers
+listed in the Registry of Registries.

-An earlier search capability, \xmlel{vg:RISearch}, is not removed
-from the schema in order to allow operators to register RI1 registries without having to support different versions of the VORegistry schema.
-Also, the type may be useful when other registry search interfaces want to define capability types of their own.
-
-RegTAP service capabilities and other additional search capabilities other than the deprecated but allowed RISearch are to be included in the registry self-identification record as an auxiliary capability as described in the IVOA note "Discovering Data Collections Within Services" \citep{note:DataCollect}, with the main capability being the OAI-PMH harvester service endpoint.
+An earlier search capability, \xmlel{vg:RISearch}, is not removed from
+the schema in order to allow operators to register RI1 registries
+without having to support different versions of the VORegistry schema.
+Also, the type may be useful when other registry search interfaces want
+to define capability types of their own.

-The concept of a searchable registry versus a publishing one as recognized by the Registry of Registries therefore now encompasses any registry implementing an IVOA standard programmatic interface beyond the interface for OAI harvesting. That is, any Registry resource with an additional capability or auxiliary capability referencing an IVOA standard and a ParamHTTP interface element to that capability.
+RegTAP service capabilities and other additional search capabilities
+other than the deprecated but allowed RISearch are to be included in the
+registry self-identification record as an auxiliary capability as
+described in the IVOA note "Discovering Data Collections Within
+Services" \citep{note:DataCollect}, with the main capability being the
+OAI-PMH harvester service endpoint.
+
+The concept of a searchable registry versus a publishing one as
+recognized by the Registry of Registries therefore now encompasses any
+registry implementing an IVOA standard programmatic interface beyond the
+interface for OAI harvesting. That is, any Registry resource with an
+additional capability or auxiliary capability referencing an IVOA
+standard and a ParamHTTP interface element to that capability.

\appendix

@@ -803,7 +920,10 @@
in the Registry.  It is unchanged from version 1.0 of this specification
and therefore does not change its version.

-Note that standards defining search interfaces may specify alternative or complementary methods of registering the services defined by them, and that auxiliary capabilities for these search capabilities may be listed within the registry record.
+Note that standards defining search interfaces may specify alternative
+or complementary methods of registering the services defined by them,
+and that auxiliary capabilities for these search capabilities may be
+listed within the registry record.

\lstinputlisting[language=XML]{VORegistry-1.0.xsd}

@@ -870,16 +990,21 @@

\begin{itemize}

-\item Corrected reference to OAI-PMH spec in registry interface description to v2.0.
+\item Corrected reference to OAI-PMH spec in registry interface
+description to v2.0.

-\item Added requirement for OAI-PMH interface to support seconds granularity, optional in the OAI-PMH 2.0 standard itself. {}
+\item Added requirement for OAI-PMH interface to support seconds
+granularity, optional in the OAI-PMH 2.0 standard itself. {}

-\item Removed requirement for VOResource version number changes to force an update of this document. {}
+\item Removed requirement for VOResource version number changes to force
+an update of this document. {}

-\item Removed the implementation-dependent requirement for searchable registries in section 2, specifically the SOAP-based services
+\item Removed the implementation-dependent requirement for searchable
+registries in section 2, specifically the SOAP-based services
based on ADQL 1.0'' and XQuery.{}

-\item Dropped the requirement on registries to not deliver any records that are OAI-PMH deleted when no temporal constraint is given.{}
+\item Dropped the requirement on registries to not deliver any records
+that are OAI-PMH deleted when no temporal constraint is given.{}

\item Added a requirement to provide VOSI endpoints.

@@ -889,7 +1014,9 @@
months only applies to the transient case; also discouraging registries
with no support of deleted records.

-\item Added recommended process for discovery of registries and their resources using the Registry of Registries, based on the Registry of Registries IVOA note
+\item Added recommended process for discovery of registries and their
+resources using the Registry of Registries, based on the Registry of
+Registries IVOA note

\item Many editorial changes across the text, mostly as a consequence of
externalizing the search interface.