Working with AWS EBS on Unix/Linux

Decided to make a quick note on how to work with AWS EBS. There are several ways. Tell how it can be done in several ways.

Working with AWS EBS on Unix/Linux (Manual)

Will show a visual example of how you can create, pronounceth volume to the created machine. To begin with, which will create partition (AWS EBS to volyum) for this login in to the AWS Console. Then, go to the EC2 section. Then, the Volume section. After that, click on “Create Volume”:

After creating a new partition, it should predacity on the target instance(the machine), it open EC2 -> ELASTIC block STORE find BLOCK “Volumes”, and then find the created partition. Click on the desired volyum right mouse button and select “Attach Volume” (you can also click on “Actions -> Attach Volume”):

Go to a server, I have it revolve Jenkins.

Create a folder:

$ mkdir /var/build/garson_home

Look what drives are in the system:

[email protected]:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 12.6 M 1 loop /snap/amazon-ssm-agent/295 loop1 7:1 0 87M 1 loop /snap/core/5145 xvda 202:0 0 100G 0 disk `-xvda1 202:1 0 100G 0 part / xvdf 202:80 0 1.5 T 0 disk [email protected]:~#

ATTENTION! Determine if the file system on the volume(volyum). New volumes are raw block devices, and you have to create a filesystem on it before you will be able to mount and use. Volumes that were restored from snapshots probably already have the file system; if you create a new filesystem over an existing filesystem, the operation overwrites your data.

Use the command “file-s” to get information about the device, for example, the file system type:

[email protected]:~# file-s /dev/xvdf /dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=aee3cf72-5a23-4ea6-9c50-af100dfc0912 (needs journal recovery) (extents) (large files) (huge files) [email protected]:~#

Now can smontirovat:

[email protected]:~# mount /dev/xvdf /var/build/garson_home [email protected]:~#

If got error:

mount: /var/build/garson_home: wrong fs type, bad option, bad superblock on /dev/xvdf, missing codepage or helper program, or other error. 

The decision — the formatting of the volumes:

# mkfs -t ext4 /dev/xvdf mke2fs 1.44.1 (24-Mar-2018) Creating filesystem with 4k blocks and 157286400 39321600 inodes Filesystem UUID: e91fac36-3163-415c-a583-b390cce264c7 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done

and then you can re-mount:

# mount /dev/xvdf /var/build/garson_home

To unmount, you should run:

# umount /var/build/garson_home

Got error when deleting folders:

rm: cannot remove '/var/build/garson_home': Device or resource busy

Fix:

[email protected]:~# losetup -d /dev/loop0 [email protected]:~# fuser -vm /var/build/garson_home USER PID ACCESS COMMAND /var/build/garson_home: root kernel mount /var/build/garson_home [email protected]:~# losetup -d /dev/loop1 [email protected]:~# fuser -vm /var/build/garson_home USER PID ACCESS COMMAND /var/build/garson_home: root kernel mount /var/build/garson_home [email protected]:~# losetup -a /dev/loop1: [66309]:62992 (/var/lib/snapd/snaps/core_5145.snap) /dev/loop0: [66309]:62995 (/var/lib/snapd/snaps/amazon-ssm-agent_295.snap) [email protected]:~# umount /var/build/garson_home [email protected]:~# 

Something like that. Everything seems logical and works, but…. the next time you restart the OS, the partition will not be automatically utchitsja to the file system for this — you need to add some actions in fstab.

To start, copy config:

# cp /etc/fstab /etc/fstab.orig

It’s not necessary, but if something goes wrong, you can revert back.

Use the command blkid to find the UUID of the device:

# blkid

Open fstab:

# vim /etc/fstab

And write:

UUID=YOUR_UUID_HERE /YOUR_MOUNT_PATH_HERE TYPE_FS_HERE defaults,nofail mount DUMP_HERE PASS_HERE

Where:

  • YOUR_UUID_HERE Is the UUID received from the output of the command blkid for your partition (Example: bc756bb8-a3cb-4c07-90b9-8827e73687c9).
  • YOUR_MOUNT_PATH_HERE Is the path to mount (Example: /var/build/garson_home).
  • TYPE_FS_HERE Is a type of mounted filesystem (E.g. xfs).
  • defaults,nofail mount Is a mount option.
  • DUMP_HERE — Used by the dump utility to decide when to do the backup. After installation, dump checks the entry and uses the value to decide whether to mount the file system. Possible values are 0 or 1. If 0, dump will ignore the file system, if 1, dump will make a backup. Most users dump is not installed, so the field should be set to 0. Example: 0.
  • PASS_HERE Is the number of the passage. The fsck utility checks the number provided in the box and decides in which order to check the file system. Possible values are 0, 1 and 2. File systems with a value 0 will not be checked by the fsck utility. At the root of the system should be the highest priority as 1, other filesystems should have priority 2. For example: 2.

The Internet is seen as some yuzayut:

YOUR_SECTION_HERE /YOUR_MOUNT_PATH_HERE TYPE_FS_HERE defaults,nofail mount DUMP_HERE PASS_HERE

Where:

  • YOUR_SECTION_HERE is the name of the section. For example: /dev/xvdf. To you can use the lsblk utility.

Well, admantium the volyum:

# umount /YOUR_MOUNT_PATH_HERE

And Trimontium all:

# mount-a

Working with AWS EBS on Unix/Linux (Terraform)

Also, you can write a Terraform module for working with AWS EBS. I already have it and it can pull:

$ git clone [email protected]:SebastianUA/terraform.git

In that repository lay all of my modules that I wrote. Of course, there is also to work with AWS EBS.

Working with AWS EBS on Unix/Linux (Python + boto3)

Wrote a script (Name of script will be ebs_volumes.py) and its contents:

#!/usr/bin/env python3 # -*- coding: utf-8 -*- import argparse import boto3 import time import re import from ec2_metadata ec2_metadata class bgcolors: def __init__(self): self.colors = { 'PURPURE': '33[95m', 'BLUE': '33[94m', 'GREEN': '33[92m', 'YELLOW': '33[93m', 'RED': '33[91m', 'ENDC': '33[0m', 'BOLD': '33[1m', 'UNDERLINE': '33[4m' } def ec2_connector(b3_client, region, prof_name): try: session = boto3.session.Session(profile_name=prof_name) ec2 = session.client(b3_client, region_name=region) return ec2 except Exception as err: print("Failed to create a boto3 EC2 client connection to:n", bgcolors().colors['RED'] + str(err), bgcolors().colors['ENDC']) return False def ec2_describe_parameters(): instance_id = ec2_metadata.instance_id availability_zone = ec2_metadata.availability_zone parameters = {'Instance_ID': instance_id, 'Availability_Zone': availability_zone} return parameters def ec2_describe_volumes(b3_client, region, prof_name): volumes_array = [] ec2 = ec2_connector(b3_client, region, prof_name) if ec2: volumes = ec2.describe_volumes(Filters=[]) for volume in volumes['Volumes']: volume_id = volume['VolumeId'] if 'Tags' in volume: tag_keys = volume['Tags'] for tag_key in tag_keys: tk = tag_key['Key'] if tk == "Name": tv = tag_key['Value'] data_tags = {'Volume_Name': tv, 'Volume_ID': volume_id} volumes_array.append(data_tags) else: data_tags = {'Volume_Name': 'None', 'Volume_ID': volume_id} volumes_array.append(data_tags) else: exit(-1) return volumes_array def ec2_create_volume(b3_client, region, prof_name, vol_name, vol_env, vol_size): ec2 = ec2_connector(b3_client, region, prof_name) if ec2: volumes = ec2_describe_volumes(b3_client, region, prof_name) availability_zone = ec2_describe_parameters()['Availability_Zone'] # availability_zone = 'us-west-2a' if not volumes: print('None, I will create a new volume!') try: new_volume = ec2.create_volume(Size=int(vol_size), Encrypted=False, AvailabilityZone='{0}'.format(availability_zone), VolumeType='gp2', TagSpecifications = [ { "ResourceType": 'volume', 'Tags': [ { 'Key': 'Name', 'Value': '{0}-ebs-{1}'.format(vol_name, vol_env) }, { 'Key': 'Environment', 'Value': '{0}'.format(vol_env) }, { 'Key': 'Orchestration', 'Value': 'py-script' }, { 'Key': 'Createdby', 'Value': 'Vitaliy Natarov' } ], }, ] ) ec2.get_waiter('volume_available').wait(VolumeIds=[new_volume['VolumeId']]) except Exception as err: print( "I can not create a [{0}] volume:n".format(vol_name), bgcolors().colors['RED'] + str(err), bgcolors().colors['ENDC']) else: circle = 0 founded_volumes = [] while circle < len(volumes): if re.search(r'{0}-ebs-{1}'.format(vol_name, vol_env), volumes[circle]['Volume_Name']): founded_volumes.append(volumes[circle]) circle += 1 if not founded_volumes: print('I will create a new volume!') try: new_volume = ec2.create_volume(Size=int(vol_size), Encrypted=False, AvailabilityZone='{0}'.format(availability_zone), VolumeType='gp2', TagSpecifications=[ { "ResourceType": 'volume', 'Tags': [ { 'Key': 'Name', 'Value': '{0}-ebs-{1}'.format(vol_name, vol_env) }, { 'Key': 'Environment', 'Value': '{0}'.format(vol_env) }, { 'Key': 'Orchestration', 'Value': 'py-script' }, { 'Key': 'Createdby', 'Value': 'Vitaliy Natarov' } ], }, ] ) ec2.get_waiter('volume_available').wait(VolumeIds=[new_volume['VolumeId']]) except Exception as err: print( "I can not create a [{0}] volume:n".format(vol_name), bgcolors().colors['RED'] + str(err), bgcolors().colors['ENDC']) else: print('Woops.... I cant create a new volume. Please use another name to it!!!!!') else: exit(-1) return ec2_create_volume def ec2_attaching_volumes(b3_client, region, prof_name, vol_name): ec2 = ec2_connector(b3_client, region, prof_name) if ec2: instance_id = ec2_describe_parameters()['Instance_ID'] volumes = ec2_describe_volumes(b3_client, region, prof_name) # # instance_id = 'i-0f6f70b38d1ccf0c1' # # volumes = [{'Volume_Name': 'test-ebs-nonprod', 'Volume_ID': 'vol-040d07848d558e1da'}, # {'Volume_Name': 'test2-ebs-nonprod', 'Volume_ID': 'vol-040d07848d558e1db'}] if not volumes: print('None') exit(0) else: circle = 0 founded_volumes = [] while circle < len(volumes): if re.search(r'{0}'.format(vol_name), volumes[circle]['Volume_Name']): founded_volumes.append(volumes[circle]) circle += 1 symbols_for_volumes = ['b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n'] circle2 = 0 while circle2 < len(founded_volumes): volume_device_name = '/dev/xvd{0}'.format(symbols_for_volumes[circle2]) try: print('I will try attach!') print(volumes[circle2]['Volume_ID'], instance_id, volume_device_name) ec2.attach_volume( VolumeId='{0}'.format(volumes[circle2]['Volume_ID']), InstanceId='{0}'.format(instance_id), Device='{0}'.format(volume_device_name)) except Exception as err: print("I can not attach [{0}] to volume [{1}] node:n".format(volumes[circle2]['Volume_ID'], instance_id), bgcolors().colors['RED'] + str(err), bgcolors().colors['ENDC']) circle2 += 1 else: exit(-1) return ec2_attaching_volumes def ec2_delete_volume(b3_client, region, prof_name, vol_name, vol_env): ec2 = ec2_connector(b3_client, region, prof_name) if ec2: volumes = ec2_describe_volumes(b3_client, region, prof_name) circle = 0 if volumes: while circle < len(volumes): if re.search(r'{0}-ebs-{1}'.format(vol_name, vol_env), volumes[circle]['Volume_Name']): volume_id = volumes[circle]['Volume_ID'] try: delete_volume = ec2.delete_volume(VolumeId='{0}'.format(volume_id)) # ec2.get_waiter('delete_complete').wait(VolumeIds=[delete_volume['VolumeId']]) except Exception as err: print( "I can not delete a [{0}] volume:n".format(vol_name), bgcolors().colors['RED'] + str(err), bgcolors().colors['ENDC']) circle += 1 else: print('I dont have needed volume name to delete! Please use another one....') else: exit(-1) return ec2_delete_volume def main(): start__time = time.time() parser = argparse.ArgumentParser(prog='python3 script_name.py -h', usage='python3 script_name.py {ARGS}', add_help=True, prefix_chars='--/', epilog="'created by Vitalii Natarov"') parser.add_argument('--version', action='version', version='v0.5.0') parser.add_argument('--bclient', '--boto3-client', dest='boto3_client', help='Set boto3 client', default='ec2') parser.add_argument('--region', dest='region', help='AWS region Set for boto3', default='us-west-2') parser.add_argument('--pname', '--profile-name', dest='profile_name', help='Set profile name', default='default') parser.add_argument('--vol-name', '--volume-name', dest='volume_name', help='Set volume name to find|attach it', default='test') parser.add_argument('--vol-env', '--volume-env', dest='volume_env', help='Set env for volume', default='nonprod') parser.add_argument('--vol-size', '--volume-size', dest='volume_size', help='Set size for volume', default=6) group = parser.add_mutually_exclusive_group(required=False) group.add_argument('--d', dest='describe', help='Describe volumes', action='store_true') group.add_argument('--describe', dest='describe', help='Describe volumes', action='store_true') group2 = parser.add_mutually_exclusive_group(required=False) group2.add_argument('--c', dest='create', help='Create volume', action='store_true', default=argparse.SUPPRESS) group2.add_argument('--create', dest='create', help='Create volume', action='store_true') group3 = parser.add_mutually_exclusive_group(required=False) group3.add_argument('--a', dest='attach', help='Attach volume(s)', action='store_true', default=argparse.SUPPRESS) group3.add_argument('--attach', dest='attach', help='Attach volume(s)', action='store_true') group4 = parser.add_mutually_exclusive_group(required=False) group4.add_argument('--del', dest='delete', help='Delete volume', action='store_true', default=argparse.SUPPRESS) group4.add_argument('--delete', dest='delete', help='Delete volume', action='store_true') results = parser.parse_args() boto3_client = results.boto3_client region = results.region profile_name = results.profile_name volume_name = results.volume_name volume_env = results.volume_env volume_size = results.if volume_size results.describe: print(ec2_describe_volumes(boto3_client, region, profile_name)) elif results.create: ec2_create_volume(boto3_client, region, profile_name, volume_name, volume_env, volume_size) elif results.attach: ec2_attaching_volumes(boto3_client, region, profile_name, volume_name) elif results.delete: ec2_delete_volume(boto3_client, region, profile_name, volume_name, volume_env) else: print(bgcolors().colors['GREEN'], 'Please add [--describe] for or describe [--create] for create or [--attach] for attach', bgcolors().colors['ENDC']) print(bgcolors().colors['RED'], 'For help, use: script_name.py -h', bgcolors().colors['ENDC']) exit(0) end__time = round(time.time() - start__time, 2) print("--- %s seconds ---" % end__time) print(bgcolors().colors['GREEN'], "============================================================", bgcolors().colors['ENDC']) print(bgcolors().colors['GREEN'], "==========================FINISHED==========================", bgcolors().colors['ENDC']) print(bgcolors().colors['GREEN'], "============================================================", bgcolors().colors['ENDC']) if __name__ == '__main__': main()

Here is such a script. To call for help, run:

$ python3 ebs_volumes.py --help

That’s all the article “Working with AWS EBS on Unix/Linux” is completed.

Source: linux-notes.org

(Visited 1 times, 1 visits today)