Disk Space and Node Allocation

[edit] Athena Storage Resources

Athena has several storage options: your home directory for small, important files; data and scratch partitions for (semi)temporary storage of large amounts of data; and locally attached SAS disk on each compute node for low-latency I/O. An explanation of the storage architecture is given in the previous section on System Architecture, and we recommend that you read that section first.

[edit] Home directory

Your home directory is located on /share/home and is accessible from all compute nodes, not just the login node. Please do not store sizable amounts of data in /share/home as it is not designed to for high throughput. Instead, use the shared data and scratch partitions detailed below.

[edit] Data and Scratch partitions

Athena has a total of 23.2TB of network-accessed storage, broken up into the following partitions. All of the following partitions are accessible from all Athena nodes, both login and compute.

/share/home     # Users' home directories (described above)
/share/data1    # FiberChannel 6.3TB -- Phys: 2.5TB, INT: 2.5TB, Astro: 1TB
/share/scratch1 # FiberChannel 1.6TB -- ALL (2 week old files wiped at 80% utilization immediately)
/share/scratch2 # FiberChannel 1.6TB -- ALL (2 week old files wiped at 80% utilization immediately)
/share/sdata1   # SATA 11TB -- Phys 2.5TB, INT 2.5TB, Astro: 6TB
/share/sdata2   # SATA 2.7TB -- CENPA 1TB, eScience 1.7TB 

The data1, sdata1, and sdata2 partitions are for longer term storage but are not backed up. You should therefore have no expectation of robustness for anything you store on these partitions. Each user (or group) has a directory located within these directories. Quotas are enabled but not enforced. If a group exceeds quota and we run out of disk space, the group will be required to delete files.

The scratch1 and scratch2 partitions are for temporary storage only, because have less redundancy than the [s]data? partitions. Anyone may use these partitions (regardless of which group they are in). However, scratch1 and scratch2 will be periodically "wiped" without warning and all files older than two weeks will be deleted. The wiping will generally take place when each partition reaches 80% utilization.

Performance measurements for FibreChannel vs. SATA arrays are given in the previous section on Athena Architecture.

[edit] Locally attached disk

Each compute node also has 123GB of locally attached SAS storage in /state/partition1. Although we encourage everyone to use the shared disk arrays, if your application is particularly sensitive to I/O latency you may get better performance from the locally attached disk If you do utilize /state/partition1, please clean up after yourself when you are done.

Files on /state/partition1 can and will be wiped and any time without warning.

[edit] Performance considerations when using Athena storage

A complete discussion on how to use the Athena shared-storage filesystems is given in the section on Athena Perfomance "Getting the most out of Athena."

[edit] Athena Computational Resources

Athena has 1 "head" or "login" node: athena0. This node is used for file transfer, compilation, and job submission. Do not run compute tasks on the head node.

Computation should be performed on the system's 128 compute nodes, each of which have 2 4-core processors. 16 of these nodes ("debug nodes") are for short-term interactive, testing, debugging jobs. The remainder ("batch nodes") are owned by one of 4 different groups:

• INT has 32 nodes (256 cores)
• Physics has 32 nodes (256 cores)
• CENPA has 16 nodes (128 cores)
• Astronomy has 32 nodes for following groups:
  • NBODY
  • LSST
  • Astronomy

By default, anyone's job can run on any batch node, regardless of ownership. However, owners can optionally exercise the right to gain immediate access to their nodes, thereby preempting other users.

The "debug" machines are all in Rack 5: 1,2,3, 14,15,16,17, 20,21,22,23,24,25,26,27,28