How to import multiple device classes to the Catalogue
======================================================

:audience:`administrators`

The Device Classes Catalogue is available on the Tango Controls web page:
http://www.tango-controls.org/.

To import multiple device classes to the catalogue from a repository please use an
`import script <https://github.com/piogor/dsc-import>`_.

Requirements
------------

- Python2.7
- Subversion
- SVN python library: :command:`pip install svn`

    .. warning::
      The library will not work if any of files in the SVN has no author defined which is the case for tango-ds
      repository on the Sourceforge. To avoid problems one can edit :file:`svn/common.py` around line 358 to have
      something like the following:

      .. code-block:: python

                author = ''
                if commit_node.find('author') is not None:
                    author = commit_node.find('author').text
- urllib2 library if it is not installed with Python itself: :command:`pip install urllib2`
- Requests library version >= 2.12, you may need to run pip with --upgrade option:
  :command:`pip install --upgrade Requests`


How-to import multiple classes
------------------------------

#. Install packages listed in `Requirements`_ if not yet installed
#. Create a folder for a local copy of a repository:
    - example: :command:`mkdir ~/tmp/local-repo`
#. Clone the import utility with git:
    - :command:`git clone https://github.com/piogor/dsc-import.git`
#. Get into cloned sources:
    - :command:`cd dsc-import`
#. Make your local branch to be sure your settings will not be overwritten by someone else.
    - :command:`git checkout -b my_local_branch`
#. Update variables in a file :file:`dsc_import_utility.py` to reflect your environment:

    .. code-block:: python

        FORCE_UPDATE = False  # when True no timestamps are checked and updates are performed
        TEST_SERVER_AUTH = False  # Set true if script is run against test server with additional authentication (webu test)
        VERIFY_CERT = False  # set this to false if running aginst test server without a valid certificate
        USE_DOC_FOR_NON_XMI = True # when True, parse documentation to get xmi conntent for device servers without XMI
        ADD_LINK_TO_DOCUMENTATION = True # when True it provides a link to documentation

        # set the following variables to point to the repositories
        LOCAL_REPO_PATH = '/home/piotr/tmp/tango-ds-repo/'  # local copy of the repository will be synced there
        LOG_PATH = '/home/piotr/tmp'  # where to log some information about import process, not used now.

        REMOTE_REPO_HOST = 'svn.code.sf.net'  # host of the SVN repository
        REMOTE_REPO_PATH = 'p/tango-ds/code'  # path within the server where the repository is located

        # if one would like to limit a search tree (useful for one device server update and/or tests)
        REPO_START_PATH = 'DeviceClasses'  # do not provide start nor end slashes

        # Tango Controls or test server address
        SERVER_BASE_URL = 'http://www.tango-controls.org/'

#. run with a command: :command:`python dsc_import_utility.py`

    .. note::
      It will ask you for your credentials for `tango-controls.org` and import/update device classes
      using provided account.

How the script works
--------------------

It does import in the following way:

- It makes a local copy  (in path defined by `LOCAL_REPO_PATH`) of a SVN repository to speed up search procedure.
- Then it searches the local copy for folders containing .XMI files. It takes into account the
  standard *branches/tags/trunk* structure. The folders where it findes .xmi files or a proper structure are listed
  as candidates to be device servers.
- Then, the list of candidates then is processed and compared (by repository URL) with content in
  the Device Classes Catalogue.

  - If there are changes or `FORCE_UPDATE` is True the catalogue is updated

    - For device server without .XMI file it looks for documentation server and tries to parse html documentation
      generated by :program:`Pogo`.

  - If there ara no changes the device server is skipped

These information you will find in a :file:`README.rst`
( https://github.com/piogor/dsc-import/blob/master/README.rst ).