Using an object storage service is often the primary choice in cloud computing. Its benefits, including ease of use and small operational overhead, are extremely appealing to cloud customers. In Google Cloud, the object storage service is provided by the suitably named Google Cloud Storage service. Show
In this article we are going to explain how to use Google Cloud Storage via the command-line tool gsutil, and other ways to interact with Google Cloud Storage. This can be also helpful for users who are getting started with the new NetApp® Cloud Volumes ONTAP for Google Cloud. To get started now, click the links below: What Is Object Storage?In object storage, files are simply referenced as objects and organized into buckets—logical namespaces that are able to hold one or more objects. While a bucket can have multiple objects, an object can only belong to just one bucket. The object storage service offered by Google Cloud is called Google Cloud Storage. Google Cloud Storage comes with built-in capabilities to manage and automate bucket lifecycle rules. In Google Cloud Storage, the availability of the data can be configured by choosing a storage class for the bucket and its objects: Standard, Nearline, and Coldline. In addition, the data location can be specified, which also impacts the availability, by choosing between region, dual-region and multi-region. Google Cloud Storage comes with built-in capabilities to manage and automate bucket lifecycle rules. Different Ways to Interact with Google Cloud PlatformUsing cloud computing resources has never been easier. The number of tools and overall methods one has at their disposal to interact with these resources is useful to a range of different users. In Google Cloud Platform, there are three different methods that can be used:
What would you use if you had multiple files to store in object storage in the cloud? The gsutil command-line tool of course! Now let’s see how to do it. How to Use the gsutil Command-Line ToolIn this section we are going to explore how the gsutil command-line tool works in practice. This tool can help with the transfer of several objects, including folders, and is also well suited for automating tasks that are not so easy to achieve via the Google Cloud Console. How to Set Up a Workspace with Google Cloud Tools1. Start by logging in to the Google Cloud Console (the web user interface) and create a new Google Cloud Platform project with billing configured and enabled. 2. Download and install the latest version of Google Cloud SDK from the official documentation website. Depending on your machine operating system, the installation process may vary slightly, but in general should be quite straightforward. 3. You can verify that the install went successfully by opening up your machine terminal and executing the command gsutil -v. You will be greeted with the version of the gsutil installed on your machine (Fig. 1). If not, please make sure to go back to steps 1 and 2 before proceeding further. 4. To start using gsutil, you first need to authenticate it with your Google Cloud Platform account. Issue the command gcloud auth login and you will be directed to your browser to authenticate. Once the process is successful, both your email and active GCP project will be displayed in the terminal (Fig. 2). Creating a Google Cloud Storage Bucket with gsutilCreating a Google Cloud Storage bucket is the first step before any files (i.e. objects) can be uploaded to Google Cloud Storage. The bucket is a virtual namespace for our objects and, as you might recall, an object can only belong to a single bucket. 1. Using your machine terminal, issue the command gsutil help mb (Fig. 2.). This help command will provide you all the syntax details of the mb (make bucket) option. You can use the help command with any of the gsutil options, which is an invaluable resource for providing information and detailed explanations on all the mandatory and optional flags. 2. Creating the Google Cloud Storage bucket is, as you might have noticed from the previous output (Fig. 3 above), quite simple. Use the command gsutil mb gs://<YOUR_BUCKET_NAME> to create it (Fig. 4). You will need to make sure that <YOUR_BUCKET_NAME> is a unique name, not just in your GCP account but in the entire global Google Cloud Platform. Adding/Removing Data from a Google Storage Bucket with gsutilThe process of adding and removing data from a Google storage bucket is quite simple and fast, with the only potential bottleneck being the speed of your internet connection. If you are used to the terminal, you may be familiar with the concept of the commands cp (copy) and rm (remove). The Google Cloud Storage command-line tool gsutil uses the exact same concept. 1. Start by changing the directory to the folder in your local machine where the files you want to upload are located. In our example case, this would be a folder in our Desktop called “photos” (Fig. 5). 2. In order to copy all the files in the current directory to the storage bucket we previously created, we will use the command gsutil cp * gs://<YOUR_BUCKET_NAME> (Fig. 5). One important aspect to note is that when performing an upload of multiple large files, using the flag -m, to perform a parallel (multi-threaded/multi-processing) copy, will significantly improve the performance. Also, if you wish to include local subfolders in your upload, you will be required to use the flag -r, i.e., to perform a recursive copy. 3. You can verify the files were properly created by using the command gsutil ls gs://<YOUR_BUCKET_NAME> and list the contents of the bucket (Fig. 5). 4. Likewise, if you wish to remove the objects from the storage bucket, you can issue the command gsutil rm gs://<YOUR_BUCKET_NAME>/* to remove all the files inside the bucket (Fig. 6). 5. Clean up the environment by deleting the Cloud Storage bucket you created, by using the subcommand rb (remove bucket), i.e., with the command gsutil rb gs://<YOUR_BUCKET_NAME> (Fig. 6). Read More on How to Use Google CloudYou've just seen how to interact with the Google Cloud Platform's console, command-line tools, and SDK’s. Now that you know how to install and authenticate with Google Cloud using your machine terminal, there's a lot more on Google Cloud and Google Cloud Storage with Cloud Volumes ONTAP. Follow the links below to learn how to: Week 1 notes — Introduction to Google Cloud When you run workloads in GCP, you use projects to organize them. You use Google Cloud Identity, and Access Management also called IM, or IAM to control who can do what. You also use your choice of several interfaces to connect. We will learn these basics this week. Projects are the main way to organize the resources in GCP. You can use them to group together related resources, normally ones that have a common business objective. Principle of least privilege There are four ways to interact with GCPs management layer When you build an application on your on-premises infrastructure, you’re responsible for the entire stack security. This would include the physical security of the hardware, the premises in which they’re housed, the encryption of the data on disk, the integrity of your network, even securing the content stored in those applications. GCP will handle the many lower layers of security, but the upper stacks remain the customer's responsibility. Google does, however, provide tools like IAM to help customer implement the policies they choose We are going to start from the bottom up. All the resources you use, whether they’re virtual machines, cloud storage buckets, tables, and big query or anything else in GCP are organized into projects. Projects can be organized into folders. Folders can contain other folders. All folders and projects used by our org can be brought together under an organization node. Projects, folders, and organization nodes are all places where the polices can be defined. Some GCP recesses let you put policies on individual resources too, like Cloud Storage buckets. Policies are inherited downwards in the hierarchy. All Google Cloud platform resources belong to a project. Projects are the basis for enabling and using GCP services — like managing APIs, enabling billing, adding, and removing collaborators, enabling other GCP services. Each project is a separate compartment and each resource belongs to exactly one. Projects can have different owners and users — they are built separately and managed separately. Each GCP project has a name and project ID that you will assign. The project Id is permanent, it’s unchangeable, and a unique identifier across all of GCP. You use project Ids in many contexts to tell GCP which project you want to work with. GCP will also assign each of your projects a unique project number that you will see displayed in different contexts. In general, project Ids are made to be readable by humans and referred to in projects. You can organize projects into folders (not required). An example would be organizing folders to represent departments, teams, applications, or environments in your org. Folders let teams easily delegate administrative rights so they can each work independently. Resources in a folder will inherit IAM policies from the parent folder. In the example above, you can put your IAM policies into folder B if both project_3 and project_4 are administered by the same team. This helps reduce errors and tedious work. Organization Nodes To use folders, you need an organization node at the top of the hierarchy. This is a place that can have central visibility on how resources are being used and policies being applied centrally. You could designate an organization policy admin so that only people with privilege can change policies. You could also assign a project creator role — a great way to control who can spend money. Once you have an organization node, you can create folders under it and create projects. Below is an example of inheritance from an organization node. One thing to keep in mind: policies implemented at a higher level in this hierarchy can’t take away access that’s granted at a lower level. Choose the correct completion: Services and APIs are enabled on a per-__________ basis. True or false: Google manages every aspect of Google Cloud Platform customers’ security. Your company has two GCP projects, and you want them to share policies. What is the less error-prone way to set this up? IAM lets admins authorize who can take action on specific resources. An IAM policy has a “who”, and “can do what”, and an “on which resource”. most of the time, to do any meaningful operations, you will need more than one permission. An example is managing instances in a project — you will need to create, delete, start, stop, and change an instance. So permissions are grouped together into a role to make them easier to manage. There are three kinds of roles in IAM: These roles are broad, you can apply them to a GCP project and they affect all resources in that project. These are the owner, editor, and viewer roles. Compute engines InstanceAdmin Role lets whoever has that role perform a certain set of actions on virtual machines These actions are:
On which virtual machines you may ask? That all depends on where the roles are applied. The example below shows that these are all defined on project_a If even finer-grained roles are needed, there are custom roles. You may be familiar with a least-privileged model in which each person in your organization has the minimum amount of privilege needed to do his or her job. One example could be — maybe I want to define an InstanceOperator Role to allow some users to start and stop Compute Engine and virtual machines, but not reconfigure them. Custom roles allow us to do that. Custom roles can only be used at the project or org levels, not folder levels. There is also the option the give access to say, a Compute Engine virtual machine, instead of a person. This is when you would use a service account. For instance, maybe you have an application running in a virtual machine that needs to store data in Google Cloud Storage, but you don’t want to let just anyone on the Internet have access to that data, only that virtual machine. So, you’d create a service account to authenticate your VM to cloud storage. Service accounts are named with an email address. But instead of passwords, they use cryptographic keys to access resources. In this simple example, a service account has been granted Compute Engine’s InstanceAdmin Role. This would allow an application running in a VM with that service account to create, modify, and delete other VMs. Service accounts need to be managed too. In addition to being an identity, service accounts are also resources. So it can have it's own IAM policies. Quiz — Resources and IAMWhen would you choose to have an organization node? When you want to apply organization-wide policies centrally. Organization nodes let you apply policies centrally. Organization nodes are optional, but if you want to define policies that apply to all the projects in your organization, having one is mandatory. Order these IAM role types from broadest to finest-grained. Can IAM policies that are implemented higher in the resource hierarchy take away access that is granted by lower-level policies? Compared to AWS IAMThese bad boys kinda do the same thing. It’s all just a mechanism to secure user authentication and permission for the cloud. Here is a chart briefly showing the differences Interacting with GCPThere are four ways to interact with GCP: the Console, the SDK & Cloud Shell, the Mobile App, and the APIs. GCP Console A web-based administrative interface. It lets you view and manage all projects and all the resources they use. GCP Console also lets you enable, disable, and explore the APIs of the GCP services. Cloud Shell A command-line interface to GCP that’s easily accessed from your browser. From Cloud Shell, you can use the tools provided by the Google Cloud Software Development kit SDK without having to install them somewhere first. SDK is a set of tools that you can use to manage your resources and your applications on GCP. These include the gcloud tool, which provides the main command-line interface for GCP products and services. There’s also gsutil which is for Google Cloud Storage and bq which is for BigQuery. The easiest way to get the SDK commands is to click the Cloud Shell button on the GCP console. You then get a command line in your web browser on a virtual machine with all these commands already installed. You can also install the SDK on your computer and is available as a docker image. Mobile There’s a mobile App for Android and iOS that lets you examine and manage the resources you’re using in GCP. It lets you build dashboards so that you can get the information you need at a glance. REST-based API Your code can use Google services in much the same way that web browsers talk to web servers. REST. The APIs name resources and GCP with URLs. Your code can pass information to the APIs using JSON. There is an open system for user login and access control. The GCP Console also lets you turn on and off APIs. Many APIs are off by default, and many are associated with quotas and limits. These restrictions can help protect you from using resources inadvertently. You can enable only those APIs you need and you can request increases in quotas when you need more resources. API ExplorerThe GCP Console includes a tool called the APIs Explorer that helps you learn about the APIs interactively. It lets you see what APIs are available and in what versions. These APIs expect parameters and documentation on them is built-in. You can try the APIs interactively even with user authentication. Google provides client libraries that take a lot of the work out of the task of calling GCP from your code. There are two kinds of libraries. The Cloud Client Libraries are Google clouds latest and recommended libraries for its APIs. They adopt the native styles and idioms of each language. On the other hand, sometimes a Cloud Client Library doesn’t support the newest services and features. In that case, you can use the Google API Client Library for your desired languages. These libraries are designed for generality and completeness. Cloud marketplaceCloud marketplace is for staring up with GCP with little to no effort. Quickly deploying software packages on GCP. pre-packaged, ready-to-deploy solutions. There’s no need to manually configure software, virtual machine instances, storage, or network settings. Although you can modify many of them before you launch if you like. Quiz — Getting Started with Google Cloud PlatformTrue or False: In Google Cloud IAM: if a policy applied at the project level gives you Owner permissions, your access to an individual resource in that project might be restricted to View permission if someone applies a more restrictive policy directly to that True or False: All Google Cloud Platform resources are associated with a project. Service accounts are used to provide which of the following? (Choose all that are correct. Choose 3 responses.) How do GCP customers and Google Cloud Platform divide responsibility for security? Which of these values is globally unique, permanent, and unchangeable, but chosen by the customer? Consider a single hierarchy of GCP resources. Which of these situations is possible? (Choose all that are correct. Choose 3 responses.) note: these are the three correct answers What is the difference between IAM primitive roles and IAM predefined roles? Which statement is true about billing for solutions deployed using Cloud Marketplace (formerly known as Cloud Launcher)? |