Welcome

User's manual for Asix 7

www.asix.com.pl

AspadTools

Program designed to process and maintain files of Y, M, D and H-type archives

Doc. No ENH7018

Version: 2014-05-08


Preface

1.1 Using AspadTools

AspadTools is designed to process and maintain files of Y, M, D and H-type archives.

It allows to:

  • archive conversion into another type (e.g. from M to D),

  • export to a text file,

  • verify and mend an archive,

  • acquire statistics concerning data size for individual archived variable trends,

  • rename or delete a variable from an archive file,

  • convert the archive time from the local time to UTC and from UTC to the local time.


Main Window

1.2 Main Window

The following figure presents the main window of AspadTools.

Fig. The Main Window of AspadTools.

zoom in >>

In this window, selected actions on chosen files can be performed.

Below the ribbon with tabs and commands there is a field that enables selection of a directory containing the archive.

The list of files can be displayed in a selective manner by choosing the appropriate filters in selection fields over the field with files. Only files related to a specific resource can be selected, while <all> means that all resources are selected. Marks of individual file types (Y, M, D, H) can also be used in the field Filter. Only files of selected types will appear in the list.

One or more files can be selected from the file list, on which specific actions, such as archive conversion into another type or statistics collection will be carried out.

An appropriate action can be initialized by means of the context-sensitive menu (after pressing the right mouse button) or by clicking on appropriate button on the ribbon.

The trace of the action is recorded in AspadTools.log file in directory the archive files are located in.


AspadTools and Archive File Versions

1.3 AspadTools and Archive File Versions

As a principle, AspadTools is designed for operation with the current ASPAD version. It means that the files created by AspadTools will be compatible with the current ASPAD version and can be incomprehensible for its former versions.

AspadTools in Asix 7 supports files from ASPAD ver. 6 (included in Asix package ver. 3 and upper). For files of Aspad 5 (included in Asix ver. 2), there is the need to convert these files to a newer versionwith the use of an older version of Aspad - e.g. from Asix ver.6.

AspadTools 8allows you to create output files:

- in the version compatible with Aspad of Asix 7

or

- in the version compatible with Aspad of Asix 3-6.

The versions are determined by the settings on the Settings tab > Settings group.

Fig. The Window of AspadTools.

zoom in >>


Running AspadTools with Parameters

1.4 Running AspadTools with Parameters

AspadTools.exe may be run in command line with the following parameters:

- Archive file name;

- parameter -U means that the program should not output messages the operator has to respond to;

- parameter -R means verification and reconstruction of the file;

- passing only the archive file name means that the file will be opened in a browser.

NOTICE: An access path to the archive file should be declared.


Renaming Files

2.1 Renaming Files

From ASPAD version 6, a new convention of file names is used.

In former versions, the name of the archive file of December 2000 would be ASPAD.AC@ - where ASPAD is a constant name and time is encoded in a 3-letter name extension.

At present, the name of such a file could be M200012xx-BL06.ahf, where the 10 characters encodes the archive type and period of a given file, .ahf is a constant extension of archived file names, and BL06 is the name of archive resource. For more detailed description of file name convention, see Asix system documentation ("Asix - Functionality and Operating Rules", 7.4. The Files Created by Programs of the Aspad Modules).

NOTICE: In order to change the convention into the new one, which is accepted for Aspad 6 and upper, you have to convert the names of files. AspadTools from the version 8 does not support such a conversion, so you should perform the operation using an older version of the program - for example AspadTools from Asix ver. 6.

Change of the file names convension with the use of an older version of AspadTools

Select one or more files created in the former file name convention and perform the action Conversion > Convert File Names. The program will ask you to enter the resource name. The name that will be assigned to the resource archived in these files in the future should be entered.

If we are going to work with one archive resource only and the archive type is not going to be changed (the M archive will remain M, H will remain H), it is enough to rename the files in order to enable ASPAD 6 to work with such files.

If more than one resource will be archived, but the archive type will not change, one of the following methods can be used:

  • make a few copies of each file, changing the name section corresponding to resource name in an appropriate way,

  • any name should be entered as the resource name, e.g. "ASPAD", and files should be subject to further processing.

Changing the archive type forces to perform advanced conversion operations.

AspadTools also enables to reverse file renaming operation. To do so select appropriate files from the list and call Conversion > Restore old file names from the menu. Name extensions corresponding to file times will be assigned to the files and the name itself will be resource name. If resource name was different from "ASPAD", the names of result files should be manually changed into "ASPAD". The operation of restoring former names to files can be performed for H and M-type archives only.


Archive Conversion

2.2 Archive Conversion

File conversion operation makes it possible:

  • converting archive type e.g. from M into D,

  • distributing variables for individual resources into separate files,

  • repacking data using improved compression methods available in ASPAD 6 and upper by conversion into the same archive type,

  • exporting data to a text file.

Conversion must be carried out very carefully. Correct results will be obtained only when the following rules are observed:

  • only files in new name convention provided by ASPAD 6 can be processes correctly; files in former conventions should be renamed first,

  • saving the result in the source file is inadmissible, it is recommended to save the result files into a new directory

  • data are written at the end of result files thus the same data cannot be written (from the same period) into the same file,

  • files must be processed in chronological order (e.g. file of August cannot be converted after file of September has been converted); if a few files are selected for processing, AspadTools orders them chronologicall by itself.

Archive file conversion is initialized with Conversion > Convert files action. After it is selected conversion window is displayed where the list of operations to be performed is defined and conversion can be started.


Conversion Window

2.3 Conversion Window

After conversion operation is selected, the following window is displayed where the list of operations to be performed is defined and conversion can be started.

REMARK: In the bottom section of the window, list of operations to be performed on each file is displayed. The list corresponds with contents of AspadToolsConvert.ini file, which is created automatically by AspadTools when defining conversion. After the first start-up, the list of operations is empty. See: 2.4 File of Conversion Operation Definition.

Pressing the Run button starts performing operations from the list for every file selected to be processed.

The list can be modified by adding new or removing old lines from it.

In order to delete a line from the list, select it and press the Remove button.

Before a new element is added to the list, define correct parameters in the fields in the upper section of the window and then press the Add button.

The meaning of most of the fields is obvious.

Advices on conversion can be found in chapter 2.5. How to Convert an Archive.

Fig. The Window of Archive Conversion with the Empty Line of an Operation to Be Performed.

zoom in >>

Fig. The Window of Archive Conversion with the Line of an Operation to Be Performed.

zoom in >>


File of Conversion Definition

2.4 File of Conversion Operation Definition

List of definitions of operations to be performed during conversion is saved in the AspadToolsConvert.ini file in a source archive directory.

Every operation is defined by means of the following text line with the use of the conversion window editor (see: 2.3 Conversion Window):

type_of_result_archive, name_of_result_resource, target_directory, [configuration_file][:variable_name]

where:

type_of_result_archive - Y, M, D or H,

name_of_result_resource - name of resource the data are to be put into,

target_directory - full access path to the directory the result files are to be put into,

configuration_file - optional definition of a file containing variable declarations (text file: *.aspad, *.asp, *.dat. with the list of variables to be converted; enter variable names in separate lines, with a result archive type set for each variable name after the comma); only variables defined in this file with the same archive type as the one given in type_of_result_archive will be put into a result file; if this file is omitted, all the variables of a source file will be processed.

variable_name - optional declaration of a variable; only the variable defined in this parameter will be put into a result file; if this parameter is omitted, all the variables of a source file will be processed.

EXAMPLE:

D, Aspad, D:\Test\ArcH\D, D:\Test\ArcH\Kociol_01.aspad

means that every archive file will be processed into archive D with the Aspad resource name and put into D:\Test\ArcH\D directory, while in the result file there will only be those variables, which archive has been declared in D:\Test\ArcH\Block_01.aspad file as D.

Fig. The Declaration of Convertion with the Parameter Declaring a File with Variables to Be Converted.

zoom in >>

Fig. The Declaration of Convertion with the Parameter Declaring a Variables to Be Converted.

zoom in >>


How to Convert an Archive

2.5 How to Convert an Archive

When beginning a conversion, a few cases should be distinguished:

  • we do not want to change archive type and all data will be related to one resource,

  • we want to change archive type, but all variables can be put into one resource,

  • we want to distribute data to various resources.

Export to a text file is a special conversion case. (See: 2.5.4 How to Export Data to a Text File)


Conversion to One Resource without Changing an Archive Type

2.5.1 Conversion to One Resource without Changing an Archive Type

If all data are to be put under the same resource name and we do not want to change the archive type, it is enough to change the resource name of the files being converted. For this purpose select out appropriate files and put a new resource name into the list of operations of the convertion window (See: 2.4 File of Conversion Operation Definition).

Fig. Parameters of Convertion to One Resource Without Changing an Archive Type.

zoom in >>


Conversion to One Resource with Changing an Archive Type

2.5.2 Conversion to One Resource with Changing an Archive Type

If distribution of archive resources or variables into various archive types will not be necessary, you should:

  • select files to be converted and call the file conversion window,

  • enter one line with the result type, resource and directory name into the list of operations (See: 2.4 File of Conversion Operation Definition),

  • carry out preset conversion operation.

REMARK It must be necessarily remembered about chronological file processing.

Fig. Parameters of Convertion to One Resource with Changing an Archive Type.

zoom in >>


Conversion to Multiple Resources

2.5.3 Conversion to Multiple Resources

If distribution of archive resources or conversion into various archive types will be necessary, you should:

• select files to be converted and call the file conversion window,

• enter one line for each archive type and configuration file with variables (to be converted into the spcific archive type) into the list of operations (See: 2.4 File of Conversion Operation Definition),

• carry out preset list of conversion operations.

REMARK It must be necessarily remembered about chronological file processing.

Fig. Parameters of Convertion to Distribute Archive Resources into Multiple Resources.

zoom in >>

Fig. Parameters of Convertion into Various Archive.

zoom in >>


How to Export Data to a Text File

2.5.4 How to Export Data to a Text File

Export to a text file is performed with the use of the command Export ANSI or Export UTF8 (for files that contain Unicode characters) available from: Conversion tab > Import/Export group.

Instead of separate text files corresponding to the individual archive files, only one file is created for each resource. For example, the file KURS.txt will be generated as a result of export of files: D20130107-KURS.ahf and D20130108-KURS.ahf. If there is missing data between files, gaps are written into the text file.

Every archived item is represented in one line in the text file and contains: variable name, time, OPC status encoded in hexadecimal format and variable value.

If there is no variable value in the line, the time represents the beginning of a gap in the archive.

If during presetting conversion parameters a file with variable parameters is given, the data are subject to decompression with the period corresponding to the variable sampling period.

In addition, the export dialog window allows you to decide whether constant values are to be extracted. Only the beginning of the constant value is saved into the file while the export operation performed without unpacking of data. When unpacking, the record is repeated according to the cycle of writting to the file. Gaps are never unpacked. This function works for the archive files saved in the new convention (version 2 available since the version 7.1 of Asix package including Aspad 8; more information - see: Architect.pdf/chm, Configuration of Archival Data). In the old convention (for Aspad 6 and 7 of the Asix package of the version 5.0 - 7.0) the sampling period can not be read, so unpacking will be possible only if you specify the file including the variable filter with the parameters of archiving period, e.g. when the file with the filter includes two lines:

Variable1, D, 1m

Variable2, D, 5m

the sampling period will be 1 minute for Variable1 and 5 minutes for Variable2 in exported data.

Fig. The Window 'Export to Text CSV File'.

zoom in >>


Variable Statistics

3.1 Variable Statistics

The action Calculate Variable Statistics (AspadTools main window > Variables tab > Statistics group) counts the number of 512-byte data blocks for all the variables archived in individual files. The statistics are created for every archival file separately and written to the end of AspadTools.log file.

The information on the number of blocks is useful when optimizing the application. It allows to determine which variables occupy most space in the archive and it is worth looking on their archiving parameters - mainly: sampling rate, accuracy of value and time recording and, possibly, archiving conditions.

Fig. The Command 'Calculate Variable Statistics'.

zoom in >>


Verification and Correction of Damaged Archive Files

4.1 Verification and Correction of Damaged Archive Files

Archive files sometimes get damaged. It usually happens so in case of unexpected operating system work termination (voltage decay, blue screen). AspadTools enables to verify archival file structure as well as to correct file structure or copy its contents into a new file with correct structure.

Fig. The Command 'Calculate Variable Statistics'.

zoom in >>

See the chapters: 4.2 Verification of the Archive File Structure and 4.3 Repair of the Archive File Structure.


Verification of the Archive File Structure

4.2 Verification of the Archive File Structure

In order to verify whether an archive is correct, select one or more files to be verified and perform the action Verify (AspadTools main window > Repair tab > Repair group).

The program will verify the file structure and communicate the verification result with a relevant message. More detailed information can be contained in AspadTools.log.

The program stores the verification result in a file and next time, if the file has not been modified, it is not verified again but only the result saved before is restored.

The program verifies the file structure only and not the data collected in it. File structure correctness does not guarantee correctness of the data contained in it but only ensures correct access to them.

In the case of incorrect file structure, such a file needs correction.


Repair of the Archive File Structure

4.3 Repair of the Archive File Structure

If damage in archive file structure has been detected or if such damage is suspected, it may be tried to be repaired. For this purpose, select one or more archive files and perform the action Verify and Repair (AspadTools main window > Repair tab > Repair group).

First of all file structure is verified (only if the file has not been verified before) and in the case of errors its contents is ordered and saved into another file in "__TMP__" temporary directory. If the operation is successful, the extension of the original file is changed into ".bak", and the temporary file replaces the original one. The contents of "__TMP__" directory are deleted.

The new file contains the same data as the original one. Only its structure is repaired so that correct access to collected data is possible. If it is suspected that the data can be the reason for problems, e.g. they contain infinite values or the value that are not numbers, the chance to delete them is to copy all the data into a new file.


Repacking Data to a New File

4.4 Repacking Data to a New File

If it is suspected that the data can be the reason of problems, e.g. they contain infinite values or the values that are not numbers, the chance to delete them is to copy all the data into a new file. In order to copy them, select one or more files to be verified and perform the action Copy Data (AspadTools main window > Conversion tab > Conversion group), declaring the result directory for rewritten file/files. The program will unpack all the data from the source file and repack them in the result file in an indicated directory.

All data with infinite values or values that are not numbers are deleted during the operation. In the result file, the data will be packed in accordance with the current compression method.

REMARK This method will make data be copied only for the variables for which information on variable type is maintained in a correct way (it is necessary to unpack data). If this information is destroyed, the data for the variable will be lost. Therefore, this method should be avoided if it is enough to correct the archive file structure and remember to keep a backup copy of the archive.

Fig. The Command of Repacking Data to a New File.

zoom in >>


Defragmentation of the Archive File Structure

4.5 Defragmentation of the Archive File Structure

Defragmentation does not apply to organizing files on a disc, as in the system defragmentation of the disc - it applies to the distribution of the data for every variable within the file. After defragmentation, the data for the variable will be included in the consecutive blocks of data within the file. This speeds up the first reading of variable data series. For example, we managed to get 2.5-fold acceleration of reading operation for the variable A000 from a demo application 'Factory' (included in the Asix package). On subsequent readings, the system uses the cashed data and defragmentation does not provide benefits.

Defragmentation leaves files BAK on the disc. These files need to be manually removed when we see that the result files are correct.

To defragment an archive file, use the action Defragment (AspadTools main window > Repair tab > Repair group).

Fig. The Command of Defragmentation of the Archive File Structure.

zoom in >>


Renaming and Deleting Variables

5.1 Renaming and Deleting Variables

Sometimes it is necessary to delete the current data series of a variable or to rename a variable in the archive file. The example can be a change of the variable type, which makes it impossible to read its current data series or a change of technological names of archived variables.

In order to rename a variable, select the archive files, in which changes are to be made, and execute the action Change Variable Name (available form AspadTools main window > Variables tab > Variables group). The program will inquire after the new and former variable name and, after pressing the button OK, it will rename variables on each of the files selected.

Fig. Running the Command of Renaiming a Variable.

zoom in >>

Deleting a variable involves changing its name into a name that does not clash with existing names. It is recommended to select such a name that would enable to find it easily if the former archive should be restored.

In case of any problems with finding the correct name for a variable, the variable statistics creation function can be used, which will generate a file with the list of variables.


Renaming of Many Variables

5.2 Renaming of Many Variables

In order to rename many variables in archive files in more easily way, the new function Change Multiple Variable Names has to be prepared (AspadTools main window > Variables tab > Variables group).

To change the names, the list of variable name translations in the form of text file compatible with the ZmianaDP program should be prepared. In the consecutive lines two names should be entered: a former name, white space, a new name. If the line contains only one name or is null, the line will be omitted.

After that one should select the files in which renaming will be performed.

Fig. Running the Command of Change Multiple Variable Names.

zoom in >>

After the function call it appears the warning with the recommendation to backup existing files, and then the dialog box with the query about the file with the list of variable name translations.

After the file name is confirmed, its contents is verified. If any errors occur (there is a variable with the name identical to one declared as a new name, e.g.), the conversion will not succeed, and the user will be informed about the error. Detailed information on errors will be written to the AspadTools.log file in the archive directory.

If this type errors do not appear, the variable names will be changed in the selected files.


Writing the List of Variables

5.3 Writing the List of Variables

The function Write List of Variables has been designed to make the creation of the list of variables for the function Change Multiple Variable Names easy.

The function may be called from: AspadTools main window > Variables tab > Statistics group.

In order to generate the file with names, one should select the files the list will be generated from.

Fig. File Selection for Variable List Creation.

zoom in >>

After the function call it appears the dialog box with the query about the file with the list of variable names. After the file name is confirmed, the list with all variables (set in alphabetical order) that are found in selected archival files is generated.

Fig. The File with the List of Variables.