Version 4.5 of jSCTDAQ is available since 20-11-2003.

Please report problems and bugs to Dave Robinson

This page contains installation instructions and a useage guide to the java application required to upload SCTDAQ test data to the SCT database. The java application can be invoked in two ways:

• By running the script DBUpload.cpp which is located in the sctdaq/macros folder of the sctdaq distribution.
• By running a perl script uploadSCTDAQ.pl which you can download from this site.

1. Installation
2. Uploading from the Rint session using DBUpload.cpp
3. Uploading using the perl script uploadSCTDAQ.pl
4. The Uploader GUI
5. Raw Data
6. Result File Processing
7. Special case - uploading Hybrid sctdaq data
8. Special case - using the '-noupload' option
9. Special case - using the '-nogui' option
10. Version List
11. Source code

Installation Instructions

1. Download the jSCTDAQ utility. This is a included within 'jDBUtils.jar'. Download it from Cambridge. Note you will download a zip file, please unzip it after download.
2. Make sure you have the Java RunTime Environment (JRE) or full Java Developers Kit (SDK) on your system, version **1.4** or greater. If you haven't already got them, they are free to download from Sun. Important: anything less than version 1.4 will not work!
3. Download the JDBC classes. This is a single file called 'classes12.zip'. Download it from the Geneva web server. Do not attempt to unzip this file!
4. Download the Geneva upload utilities. This is a single file called 'SCTTEST_3_nn_P.jar', where nn is the latest version number (18 or greater). Download it from the the Geneva web server.
5. Set your CLASSPATH environment variable to include all 3 files, ie jDBUtils.jar, classes12.zip and SCTTEST_3_18_P.jar.
6. And thats it! Once you have downloaded the above files, and installed Java 1.4 SDK or JRE, and set your CLASSPATH environment variable, do the following:
   • Type 'java UploadTestData' at your DOS prompt or console (NOT from within Rint!). A dialog box should appear, in which case click 'Cancel' to close it. If the dialog box fails to appear, you have set your classpath incorrectly (it does not contain the file SCTTEST_3_18_P.jar)
   • Type 'java DBUpload/uploadSCTDAQ' to verify if all works ok. If not, it is likely that your CLASSPATH environment variable is not correctly set, or perhaps your PATH environment variable does not include the path the java.exe.

Uploading from Rint using DBUpload.cpp

From Rint, the first time you upload, you should type

.x DBUpload.cpp(optional argument)

and then in subsequent requests to upload in the same Rint session, just type

DBUpload(argument)



You can upload in 3 ways:

• DBUpload() - uploads ALL SCTDAQ tests of ALL modules performed in the CURRENT day.
• DBUpload("20220330nnnnnn") - upload ALL tests of module 20220330nnnnn, performed on ANY days
• DBUpload("20220330nnnnnn_yyyymmdd.txt") - upload all tests contained in the specified result file

Uploading using the perl script uploadSCTDAQ.pl

Download the perl script and check that the $RESULTS_DIRECTORY and $UPLOADS_DIRECTORY variables in the first few lines of the script are set exactly as in sctdaq/macros/parameters.h

You can run the script with no argument or up to 3 arguments:

• the result filename or the serial number
• a directory containing the result file(s) IF DIFFERENT FROM THE DEFAULT RESULTS_DIRECTORY
• the word 'hybrid' to flag that the result file contains hybrid data, not module data.
• optional flag '-noupload' to prevent the actual upload
• optional flag '-nogui' to prevent the upload AND prevent appearance of the GUI

The Uploader GUI

DBUpload invokes a java GUI (graphical user interface) that lists the tests awaiting upload in a spreadsheet:

Note that tests are only listed if they have not been uploaded before, and if the test is recognised by the SCT database. If you want to quit the application before proceeding further, just click on the 'GoAway' box at the top right corner.

You can inspect the individual test data by selecting a row to highlight a particular test, then the 'Inspect' button becomes enabled:


You can then view the data by clicking the 'Inspect' button. Note: the data has been processed and will not correspond exactly to the data in the original result file:


If you do not want to upload a specific test (eg it was a test or rubbish data), select (highlight) the test in the spreadsheet, then click 'Delete'. The test is removed from the spreadsheet, and removed from the upload queue. But note the test is not deleted from the original result file - if you re-run the application on the same result file(s) at a later date, the test will appear again in the spreadsheet.

When you decide to upload all the tests, click the 'upload' button and enter your institute username and password:


Then click on 'Enter' and all the tests will be uploaded. A window opens allowing you to monitor progress of the upload... the upload may take some time depending on the number of tests and the state of the network:

Raw Data

All raw data files are zipped by the java application before upload. There are two reasons for this:

1. Some raw data files (the RC files for 3ptGain and ResponseCurve tests) are too big for the database (limit is 32k - files are 44k) and need to be compressed before upload
2. For the trim scan tests, we want to upload two raw data files (the .trim and .mask files) but only one raw data file is permitted per test. Therefore the two files are zipped into one zip archive before upload.

Result File Processing

There is a result file of the form 'serialno_date.txt' (eg 20220330200011_20020927.txt) created and updated by SCTDAQ whenever a test is performed. The file contains upload data for the SCT database. The upload routine provided by Geneva expects a separate upload file for each test, and expects the files to conform to a specific format. The Cambridge java application performs the following actions:

1. Reads in the result file(s) specified by the DBUpload macro from the RESULTS_DIRECTORY (or from the ARCHIVE_DIRECTORY if such an environment variable is defined)
2. Verifies if the institute and usernames are valid by lookup from database.
3. Prompts the user to select a valid institute name (or username) if the institute name (or username) are invalid.
4. Verifies hybid temperature - if set to -128C, resets it to 0C.
5. Splits file(s) into separate tests - each split is flagged by the %NewTest keyword
6. Changes all spaces and tab combinations to a single white space
7. Changes any instance of 'ModIVScan' to 'DetModIV'
8. Replaces all instances of RESULTS_DIRECTORY with ARCHIVE_DIR, if ARCHIVE_DIR env variable exists and is a valid directory
9. Checks validity of various syntax
10. If test is a trim scan, confirms that .trim and .mask files exist, and zip them to a single zip file
11. Checks any raw data files (of form xxxx.txt) exist, and zip them
12. Corrects file format for scan data (upoloader requires a full-stop for each non-used scan point)
13. Checks if each test has been uploaded before by lookup from the database. If it has not been uploaded, the test is listed in a spreadsheet
14. Allows the user the inspect each test after the above processing steps
15. Allows the user to delete a test from the spreadsheet (ie remove the test from the queue to be uploaded)
16. If the user wants to upload the tests listed in the spreadsheet (ie clicks on 'Upload'), each test is written as a separate 'database file' in the UPLOAD_DIRECTORY, and all raw data files are zipped and written to the UPLOAD_DIRECTORY. The Geneva upload routine is then invoked to upload all the files in one step - a progress window opens allowing the user to view progress.
17. For each test for which the upload was successful, the database file is removed (by Geneva application) from the UPLOAD_DIRECTORY and placed into a new subdirectory. If the upload of any number of files was NOT successful, a dialogue box will alert you that something went wrong. If any database file failed to upload, a log file (ending in .log) remains in the UPLOAD_DIRECTORY which should indicate why the upload failed.
18. All zip files are deleted from the UPLOAD_DIRECTORY after the upload is completed.

Uploading Hybrid Data

To upload hybrid data, you MUST use DBUpload_Hybrid.cpp instead of DBUpload.cpp. Please download DBUpload_Hybrid.cpp and put it in your macros directory.
Alternatively, if using the perl script, run it with the 'hybrid' argument, eg
'perl uploadSCTDAQ.pl 202203300002 hybrid').

Barrel hybrids are a special case because the actual database serial number for the hybrid is different from the serial number barcoded on the module. Eg if the module serial number is 20220330200011, then the asic-stuffed hybrid serial number is 20220338200011. Therefore the uploader must substitute all instances of 20220nn02nnnnn with 20220nn82nnnnn before upload. Whether the hybrid is a barrel or endcap hybrid, the uploader will generate warning messages if you use DBUpload_Hybrid and the device is not labelled as a something_Hybrid (eg 'Barrel_Hybrid' or 'Endcap_Hybrid'). To label the device as a hybrid, list it as such in the last column of D:/sctvar/config/st_system_config.dat
Click here to see an example.

Using the -noupload' Option

The java upload application works fine for most people, but if your application fails to upload properly or 'freezes' during the upload, then you can use the -noupload option.

• if uploading from the perl script, use the -noupload' argument
• If uploading module data using DBUpload.cpp, change the following lines:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  to:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s -noupload"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  
• If uploading hybrid data using DBUpload_Hybrid.cpp, change the following lines:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s hybrid"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  to:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s hybrid -noupload"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  

• Change to the upload directory (eg 'cd upload_directory')
• type 'java UploadTestData *.sdb username password'

Using the -nogui' Option

If you use the '-nogui' option, the upload files (and associated raw data files in zip format) are created in your upload directory immediately without being prompted by the graphical user interface, ie the spreadsheet gui will not open. You will not have the opportunity to inspect or remove tests from your spreadsheet, so only use this option if you know that you want to upload all tests.

• if uploading from the perl script, use the -nogui' argument
• If uploading module data using DBUpload.cpp, change the following lines:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  to:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s -nogui"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  
• If uploading hybrid data using DBUpload_Hybrid.cpp, change the following lines:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s hybrid"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  to:
  

    sprintf(line,"java DBUpload/uploadSCTDAQ %s %s %s %s hybrid -nogui"
            ,RESULTS_DIRECTORY,UPLOAD_DIRECTORY,archive,f_stub);
  

  
  

• Change to the upload directory (eg 'cd upload_directory')
• type 'java UploadTestData *.sdb username password'

Version list

• Version 4.5 released 20/11/2003
• Put inverted commas around 'Version', 'Host' and 'Time' if they are missing
• Version 4.2 released 19/11/2003
• Minor bug fix
• Version 4.1 released 17/11/2003
• Strip out any invisible characters at the end of each line
• allow serial number to be entered as a regular expression instead of explicitly as a 14 digit number, eg "2022033020000." will upload module 20220330200001 to 20220330200009
• if DUT is missing, insert the SCTDB device description, eg 'bmMODULE'
• Version 3.9 released 19/09/2003
• Strip out the negative sign for the current (if present) in DAQ_DCS header
• Version 3.8 released 17/09/2003
• Fix bug when using archive dir on linux machine, when sctdaq was run on windows machine
• Version 3.7 released 04/07/2003
• Strip out any blank lines from result files (bug in sctdaq v3.40 for cold IVs)
• Version 3.3 released 06/06/2003
• Application filefilter now picks up optional '_runno' appended to result filename.
• Version 3.2 released 21/05/2003
• New option '-nogui' to generate upload files immmediately without being prompted by the GUI.
• Version 3.1 released 13/05/2003
• Bug fix - tests apparently removed by 'Delete' button might still have been uploaded!
• Version 3.0 released 13/05/2003
• Update for compatibility with Geneva DB upgrade to ORACLE 9i
• Version 2.5 released 23/04/2003
• Introduce '-noupload' option to prevent uploads, and allow the user to do it manually
• Release perl script uploadSCTDAQ.pl
• Version 2.3 released 09/04/2003
• Introduce Validation of serial number
• If serial number not registered in database, the upload is prevented
• If serial number is an endcap flex, look up the corresponding module (or hybrid) serial number and substitute it. If invoked from DBUpload.cpp, the module serial number is substituted, but if invoked from DBUpload_Hybrid.cpp, the hybrid serial number is substituted. The upload is prevented if the flex is not yet assembled in the database.
• DBUpload_Hybrid.cpp now works for both barrel and endcap uploads of hybrid data.
• Version 2.1 released 26/03/2003
• If both T0 and T1 are -128, prompt user to enter the temperatures manually. If subsequent tests have the same invalid temperatures, only reprompt if the run number changes or the results file changes
• Version 2.0 released 21/03/2003
• Abandon bookkeeping using local files. Instead lookup firectly from database for previous uploads of signature serialno_testname_runno_date_time from your institute
• Verify your institute name by lookup from database. If your intitute name is invalid, the user is prompted to select their institute by menu.
• Verify your username by lookup from database. If your username is invalid, the user is prompted to select their username by menu.
• Verify temperatures. If set to 128 (or -128), set to zero. Otherwise upload is rejected Geneva utility.
• Version 0.7 released 19/03/2003
• Check for institute set to UNKNOWN
• Version 0.5 released 11/02/2003
• Bug fix: potential thread synchronisation problem during upload
• Upload ROD configuration xml file (if present) as raw data to Timewalk test
• Substitute INEFFICIENT defect name with INEFF
• Version 0.4 released 16/01/2003
• Database upload file names changed to a<# of millisecs>.sdb, to ensure that the tests are uploaded in the same sequence as listed in the result file. This ensures that the database test number increases in the same order as the tests in the result file.
• Version 0.3 released 07/01/2003
• New DBupload_Barrel_Hybrid.cpp for uploading barrel hybrid sctdaq data
• Substitute serial numbers 20220nn02nnnnn -> 20220nn82nnnnn if launched from DBUpload_Barrel_Hybrid.cpp, and generate warnings if DUT not set to Barrel_Hybrid
• Version 0.2 released 03/01/2003
• Upload all raw data as zip files.

Source Code

If you are a software developer within the SCT community working on database issues, you are welcome to the source code. This link is password protected - ask me if you don't know it.