'terraform copy/upload files to aws ec2 instance
We have cronjob and shell script which we want to copy or upload to aws ec2 instance while creating instance using terraform.
we tried
- file provisioner : but its not wokring , and read this option does not work with all terraform version
provisioner "file" {
source = "abc.sh"
destination = "/home/ec2-user/basic2.sh"
}
- tried data template file option
data "template_file" "userdata_line" {
template = <<EOF
#!/bin/bash
mkdir /home/ec2-user/files2
cd /home/ec2-user/files2
sudo touch basic2.sh
sudo chmod 777 basic2.sh
base64 basic.sh |base64 -d >basic2.sh
EOF
}
tried all option but none of them working.
could u please help or advise .
I am new to terraform so struggling on this from long time.
Solution 1:[1]
somehow in corporate domain none of the options worked. but finally we were able to copy /download files using s3 bucket.
create s3.tf to upload this files basic2.sh
resource "aws_s3_bucket" "demo-s3" {
bucket = "acom-demo-s3i-<bucketID>-us-east-1"
acl = "private"
tags {
Name = "acom-demo-s3i-<bucketID>-us-east-1"
StackId = "demo-s3"
}
}
resource "aws_s3_bucket_policy" "s3_policy" {
bucket = "${aws_s3_bucket.demo-s3.id}"
policy = <<EOF
{
"Version": "2009-10-17",
"Statement": [
{
"Sid": "Only allow specific role",
"Effect": "allow",
"Principal":{ "AWS": ["arn:aws:iam::<bucketID>:role/demo-s3i"]},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::acom-demo-s3i-<bucketID>-us-east-1",
"arn:aws:s3:::acom-demo-s3i-<bucketID>-us-east-1/*"
]
}
]
}
EOF
}
resource "aws_s3_bucket_object" "object" {
bucket = "acom-demo-s3i-<bucketID>-us-east-1"
key = "scripts/basic2.sh"
source = "scripts/basic2.sh"
etag = "${filemd5("scripts/basic2.sh")}"
}
and then declared file download portion in other tpl file.
aws s3 cp s3://acom-demo-s3i-<bucketID>-us-east-1/scripts/basic2.sh /home/ec2-user/basic2.sh
Solution 2:[2]
When starting from an AMI that has cloud-init installed (which is common in many official Linux distri), we can use cloud-init's write_files module to place arbitrary files into the filesystem, as long as they are small enough to fit within the constraints of the user_data argument along with all of the other cloud-init data.
As with all cloud-init modules, we configure write_files using cloud-init's YAML-based configuration format, which begins with the special marker string #cloud-config on a line of its own, followed by a YAML data structure. Because JSON is a subset of YAML, we can use Terraform's jsonencode to produce a valid value[1].
locals {
cloud_config_config = <<-END
#cloud-config
${jsonencode({
write_files = [
{
path = "/etc/example.txt"
permissions = "0644"
owner = "root:root"
encoding = "b64"
content = filebase64("${path.module}/example.txt")
},
]
})}
END
}
The write_files module can accept data in base64 format when we set encoding = "b64", so we use that in conjunction with Terraform's filebase64 function to include the contents of an external file. Other approaches are possible here, such as producing a string dynamically using Terraform templates and using base64encode to encode it as the file contents.
If you can express everything you want cloud-init to do in a single configuration file like the above then you can assign local.cloud_config_config directly as your instance user_data, and cloud-config will should recognize and process it on system boot:
user_data = local.cloud_config_config
If you instead need to combine creating the file with some other actions, like running a shell script, you can use cloud-init's multipart archive format to encode multiple "files" for cloud-init to process. Terraform has a cloudinit provider that contains a data source for easily constructing a multipart archive for cloud-init:
data "cloudinit_config" "example" {
gzip = false
base64_encode = false
part {
content_type = "text/cloud-config"
filename = "cloud-config.yaml"
content = local.cloud_config_config
}
part {
content_type = "text/x-shellscript"
filename = "example.sh"
content = <<-EOF
#!/bin/bash
echo "Hello World"
EOT
}
}
This data source will produce a single string at cloudinit_config.example.rendered which is a multipart archive suitable for use as user_data for cloud-init:
user_data = cloudinit_config.example.rendered
EC2 imposes a maximum user-data size of 64 kilobytes, so all of the encoded data together must fit within that limit. If you need to place a large file that comes close to or exceeds that limit, it would probably be best to use an intermediate other system to transfer that file, such as having Terraform write the file into an Amazon S3 bucket and having the software in your instance retrieve that data using instance profile credentials. That shouldn't be necessary for small data files used for system configuration, though.
It's important to note that from the perspective of Terraform and EC2 the content of user_data is just an arbitrary string. Any issues in processing the string must be debugged within the target operating system itself, by reading the cloud-init logs to see how it interpreted the configuration and what happened when it tried to take those actions.
[1]: We could also potentially use yamlencode, but at the time I write this that function has a warning that its exact formatting may change in future Terraform versions, and that's undesirable for user_data because it would cause the instance to be replaced. If you are reading this in the future and that warning is no longer present in the yamldecode docs, consider using yamlencode instead.
Solution 3:[3]
I used provisioner "file" just for that, no issues...
but you do have to provide a connection:
resource "aws_instance" "foo" {
...
provisioner "file" {
source = "~/foobar"
destination = "~/foobar"
connection {
type = "ssh"
user = "ubuntu"
private_key = "${file("~/Downloads/AWS_keys/test.pem")}"
host = "${self.public_dns}"
}
}
...
}
here are some code examples:
https://github.com/heldersepu/hs-scripts/blob/master/TerraForm/ec2_ubuntu.tf#L21
Solution 4:[4]
Here is a bit simpler example how to use write_files with cloud-init as described by @martin-atkins
Content of the templates/cloud-init.yml.tpl
#cloud-config
package_update: true
package_upgrade: true
packages:
- ansible
write_files:
- content: |
${base64encode("${ansible_playbook}")}
encoding: b64
owner: root:root
path: /opt/ansible-playbook.yml
permissions: '0750'
runcmd:
- ansible-playbook /opt/ansible-playbook.yml
Content of the main.tf file:
data "template_file" "instance_startup_script" {
template = file(format("%s/templates/templates/cloud-init.yml.tpl", path.module))
vars = {
ansible_playbook = templatefile("${path.module}/templates/ansible-playbook.yml.tpl", {
playbookvar = var.play_book_var
})
cloudinitvar = var.cloud_init_var
}
}
It is possible to use variable interpolation for cloud-init and ansible-playbook templates both
Solution 5:[5]
You have to use file provisioner with connection details to the ec2 instance. A sample config would look like this:
provisioner "file" {
source = "${path.module}/files/script.sh"
destination = "/tmp/script.sh"
connection {
type = "ssh"
user = "root"
password = "${var.root_password}"
host = "${var.host}"
}
}
You can use username / password, private key or even a bastion host to connect. For more details https://www.terraform.io/docs/provisioners/connection.html
Solution 6:[6]
This worked for me:
resource "aws_instance" "myapp-server" {
ami = data.aws_ami.ubuntu.id
instance_type = xx
subnet_id = xx
vpc_security_group_ids = xx
availability_zone=xx
associate_public_ip_address = true
key_name = xx
user_data = file(xx)
connection {
type = "ssh"
host = self.public_ip
user = "ubuntu"
private_key = file(xx)
}
provisioner "file" {
source = "source-file"
destination = "dest-file"
}
}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | pratik |
| Solution 2 | Martin Atkins |
| Solution 3 | |
| Solution 4 | Roman Shishkin |
| Solution 5 | Jeevagan |
| Solution 6 | BBM |