What is Terraform? ：

Terraform is an open source resource orchestration tool launched by HashiCorp around 2014. Currently, almost all mainstream cloud service providers support Terraform, including Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, Azure, Baidu Cloud, etc. Many companies currently build their own infrastructure based on terraform.

Background: In the traditional operation and maintenance model, the launch of a business requires preparation stages such as equipment procurement, machine installation, network environment construction, and system installation. With the rise of cloud computing, major public cloud vendors have provided very friendly interactive interfaces. Users can use a browser to purchase various cloud resources on demand and quickly build a business architecture. However, with the continuous expansion of business architecture, the scale and types of cloud resource procurement are also increasing. When users need to quickly purchase a large number of different types of cloud resources, the large number of interactive operations between cloud management pages actually reduces the efficiency of cloud resource procurement. It takes about 20 minutes or even longer to initialize a classic VPC network architecture on the Alibaba Cloud console, from creating VPC, switch VSwitch to creating NAT gateway, elastic IP, and then configuring routing. At the same time, the non-replicability of work results leads to repeated work in cross-region and cross-cloud platform scenarios.

In fact, for business operation and maintenance personnel, they only care about the configuration of resources, not the steps to create these resources. Just like drinking coffee, you only need to tell the waiter what to drink and whether to add ice. If there is a complete cloud resource procurement list, this list clearly records the types, specifications, quantities and relationships between the cloud resources to be purchased, and then completes the purchase with one click, and when business needs change, you only need to change the list to quickly change cloud resources, then the efficiency will be greatly improved. In cloud computing, this is called resource orchestration. At present, many cloud platforms also provide resource orchestration capabilities, such as Alibaba Cloud’s ROS and AWS’s CloudFormation.

Define cloud resources, services or operation steps in the form of code in the template, and use the orchestration engine to realize the automated management of resources. This is Infrastructure as Code (IaC for short), which is also the most efficient implementation mode of resource orchestration. However, multiple cloud orchestration services bring high learning costs, inefficient code reuse, and complex multi-cloud collaborative workflows. Each service is limited to managing its own single cloud platform, and cannot meet the unified management of multiple cloud platforms and multiple levels (such as IaaS, PaaS) resources. How to solve the above problems? Can we use a unified orchestration tool and a common set of syntax to achieve unified management of multiple clouds including Alibaba Cloud? So Terraform was born at this time to solve these problems.

Terrafrom functions and roles:

Functional points

  • IaC: infrastructure as code, use code to manage infrastructure
  • Execution plan: display the operations performed when terraform apply
  • Resource graph: build a graph of all resources
  • Change automation: based on the execution plan and resource graph, you can clearly know the content and order to change Summary: Terraform is used for initialization of various infrastructure resources, supports multiple cloud platforms, and supports third-party service docking

Role

  • Use APIs from different providers and package them into Terraform’s standard code structure
  • Users do not need to understand the API details of each cloud computing vendor, which reduces the difficulty of deployment

Terraform architecture

Terraform itself is a plug-in-based architecture with strong scalability, which can facilitate programmers to expand Terraform. Terraform can be logically divided into two layers, the core layer (Terraform Core) and the plug-in layer (Terraform Provider).

Core layer

The core layer is actually the command line tool of terraform. It is developed in go language and is responsible for:

  1. Read .tf configuration and replace variables
  2. Resource status file management
  3. Analyze resource relationships and draw maps
  4. Dependency map and create resources Create resources according to dependencies; for resources without dependencies, they will be created in parallel (default 10 parallel processes), which is why Terraform can manage cloud resources efficiently and quickly.
  5. Call the plug-in layer with RPC

Plug-in layer

The plug-in layer is also developed in go language. Terraform has more than 250 different plug-ins, which are responsible for:

  • Accept RPC calls from the core layer
  • Provide the execution of a specific service

There are two types of plug-in layers:

Provider

Provider is responsible for integration with external APIs. For example, Alibaba Cloud Provider provides the function of creating, modifying, and deleting cloud resources in Alibaba Cloud. This plug-in is responsible for interacting with the interface of Alibaba Cloud API and providing a layer of abstraction so that programmers can orchestrate resources through terraform without understanding the details of the API. It is responsible for:

  • Initialization and external API communication
  • External API authentication
  • Defining the relationship between cloud resources and external services

For example, common providers:

1
2
3
4
5
6
7
8
9

Alibaba Cloud: https://github.com/aliyun/terraform-provider-alicloud
Baidu Cloud: https://github.com/baidubce/terraform-provider-baiducloud
Tencent Cloud: https://github.com/tencentcloudstack/terraform-provider-tencentcloud
Huawei Cloud: https://github.com/huaweicloud/terraform-provider-huaweicloud
ucloud: https://github.com/ucloud/terraform-provider-ucloud
qingcloud: https://github.com/yunify/terraform-provider-qingcloud
AWS: https://github.com/hashicorp/terraform-provider-aws
Azure: https://github.com/terraform-providers/terraform-provider-azurerm
GoogleCloud: https://github.com/hashicorp/terraform-provider-google

Provisioner

Provisioner is responsible for executing some scripts after resource creation or deletion. For example, Puppet Provisioner can download, install, and configure Puppet agent on cloud virtual machine resources after they are created.

For easier understanding, I found a component architecture diagram on the Internet to briefly explain the location of each component:

terraform architecture diagram

For daily operations of terraform, I drew a basic workflow flowchart as follows:

terraform operation flowchart

Terraform keyword explanation:

Declarative language (HCL):

Terraform is written in HashiCorp Configuration Language. HCL is declarative, that is, programmers use HCL to describe what the entire infrastructure should look like, and then leave the specific implementation work to Terraform. Programmers do not need to understand the specific steps and details of the implementation, nor do they need to understand how terraform connects to the API of the cloud service provider. Terraform will automatically download the corresponding Provider and Provisioner based on the code to take charge of the specific steps and details. The corresponding to the declarative style is the imperative style. Imperative languages ​​are executed in steps, and the order is very important. Executing imperative languages ​​on fixed inputs will produce fixed outputs. There is no difference between declarative and imperative languages, but in the field of cloud resource orchestration, declarative languages ​​are easier to implement. The cloud resource orchestration tools we see in daily life are all declarative, including AWS CloudFormation, Azure Resource Template, and Google Cloud Deoplyment Manager. If you implement resource orchestration on Tencent Cloud by calling Tencent Cloud API, it is usually imperative.

Resource state file (state)

After Terraform is initialized, a state file will be generated. The state file records the time of the most recent operation, the relevant attributes of each resource, the current value of each variable, the version of the state file, and so on.

The next time you operate, terraform will first update the current state file with the state on the cloud service provider to find out whether there are any resources that have been deleted or changed, and then decide which resources need to be deleted, updated, or created based on the .tf file. After the operation is completed, a state file will be regenerated.

Terraform backend

The integrity of resource status files is more important. For these files, we need to at least automatically lock them at the beginning of the operation until the end of the operation, so that others cannot change them; in addition, we need to track resource version changes; and perform access control on sensitive information in resource files.

Therefore, backend is closely related to how resource status files are read, stored, and locked, and how terraform apply is executed.

Terraform uses the local backend by default, that is, the status files will be stored in the current directory, and the execution of terraform code is also run in the local virtual machine. This is no problem for cloud resources managed by one person, but when the number of team members increases, everyone may have their own workbench, but a common place is needed to store resource status files. This is when remote storage can be used. Currently, terraform supports a variety of remote storage backends, including AWS s3, Hashicorp Consul, etcd, Terraform Cloud, and terraform Enterprise Edition, etc. These remote backends provide remote storage and locking of status files. Among them, terraform Enterprise Edition provides remote running of terraform, as well as some other enterprise-level features.

Terraform module

Terraform module is to write some highly reusable code into a module for others to use. The module consists of input parameters, output parameters and main logic. This is very similar to the function in traditional programming languages. Terraform provides a public module registry. After the module is written, as long as it meets the specifications, it can be published to the module registry for everyone to use. https://registry.terraform.io/