About Me!

This blog is about my musings and thoughts. I hope you find it useful, at most, and entertaining, at least.

Résumé [PDF]

Other Pages

Quotes

Links

Oak Island

Items for Sale

Presence Elsewhere

jim@jimkeener.com

del.icio.us

Twitter

Facebook

LinkedIn

GitHub

BitBucket

Keybase.io

TDF -- Text Data File rev 3

Date: 2013-10-11
Tags: computers protocols data programming

Format

Files and text fields are in UTF8, unless otherwise specified

Delimiters (,) can be escaped by:

, $%
; %$
$ $$
% %%

Headers

Headers will be formated as RFC 2616 Headers (HTTP/1.1 header syntax). Headers should be sorted in lexical order.

Field definitions

Fields definitions contain a Name, Type, and Encoding as key:comma-separated-lists sets and separated by semicolons.

Name must correspond to a field name in the first line of the data.

Optional attributes, such as Description may be included.

Example:

Field: Name: val; Type: Int; Encoding: Dec, UTF8;

Alternatively:

Field: Name: val;
       Type: Int;
       Encoding: Dec, UTF8;

Metadata

Header value Example Comment
Digest digest hex-hash sha1 yaUxx5mrIRyXNdovreYa/PFh0PE= Calculated without this line and without expanding headers (they should each be on a single line)
Last-Modified ISO 8601 date 2013-08-06T08:52:00EST
Signature user;key fingerprint;signature
Jim <jim@example.com>;f642a8d2552281d792b52a17cbe79f3163b296f3;MIGHAkER9CmV5WJPB3hnk9eD31oqhAKWTsXVKubdIffMM9ocjU667p5yDh8xrOuOx0T8xx2NTQgmnDgsrPaXLK8WiMEaaQJCAYn2TwWkSVpgTM7oFg3O6r9ZTSRTnqZhxyk3g7O1SDHcqxohBREITiMsIFFNjv6m6sj/M8e4ndlaHZVgv5J/T+NR
Because of their size, EC keys are useful for this. Calculated without this line.
Author user
Jim <jim@example.com>
Description Description This data is awesome!
Source URL or description of the source of the data (where to go to find out more or who made it. May be repeated
http://example.com/data
or
Jim's Lab @ HisHouseU

Types

Types may be chained, e.g. Bin32,Float

Type Comment
Bin Arbitrary binary data
Integer
Float
UUID UUID
Text Text
Time Date and/or Time
CI Field represents the confidence interval for the field defined in a For field in the header definition. Field definition must also have “Offset: Min” or “Offset: Max”. Field definition must also contain a p-value field containing the p-value for this CI (0.05 => 95% interval)
Geometry Stores Geometry types

Formats

Fomat Comment Types
Dec Decimal (default) Int, Float, CI
Hex Hex encoded/Base-16 Encoded Bin, Int
B32 Base-32 Encoded Bin
B36 Base-36 Encoded Bin
B58 Base-58 Encoded Bin
B64 Base-64 Encoded Bin
B85 Base-85 Encoded Bin
UU UUEncoded Bin
XX XXEncoded Bin
UTF8 UTF-8 encoded text (Default) All
ASCII ASCII Text All
Latin1 Latin1/ISO 8859/ Text All
WKT Well-Known Text Geometry
WKB Well-Known Binary Geometry
UDT Unix Date Time Time
UMT Unix Date Time, Miliseconds /Javascript Time Time
UNT Unix Date Time, Nanoseconds Time
EDT Excel Date Time Time
WFT Windows File Time Time
ISO8601 ISO 8601 format Time


Author: Jim
Description: This data was collected with a Blah Blah Spectrometer. The procedure can be found at http://example.com/proc
Digest: sha1 583816a652dcf8365cceabfc4945c35a84e1614c
Field: Name: Abs_ci_max; Type: CI, Float; Encoding: Dec; For: Absorption; Offset: max; p-value: 0.05
Field: Name: Abs_ci_min; Type: CI, Float; Encoding: Dec; For: Absorption; Offset: min; p-value: 0.05
Field: Name: Absorption; Type: Float; Encoding: Dec; Description: Absorption at 520cm-1$% over 4 experiments
Field: Name: Time; Type: Int; Encoding: Dec; Description: Seconds from starting
Last-Modified: 2013-10-04T08:52:00EST

Time, Absorption, Abs_ci_min, Abs_ci_max
0,0.0,0.0,0.0
10,1.0,0.0,3.0
15,4.0,2.0,5.0
20,9.0,6.0,12.0
23,14.0,11.0,18.0

Note: Windows File Time is in 100-nanosecond ticks since 12:00 A.M. January 1, 1601 UTC