Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Thursday, February 02, 2023

How to find Private Network Load Balancer (NLB) ALBv2 IP4 addresses for adding to security group to allow health checks to pass

 Use case:

My Target Groups attached to my Load Balancer route to a private internal port that is not on the security group white listing.   Due to Network Load Balancer's use of the EC2 connected Security Group. I don't want to whitelist the private internal port to the internet for possible bypass.

This can be done manually in the aws console by finding the ENI by looking up the ELB (type)/(name)/(random string) getting the private ip4 and then adding them manually to the security group.


But this defeats the purpose of having a cloudformation script that automatically stands up everything to work.

in this example we want to get the private ip4 address of a ELBv2 (network). Full working example: https://github.com/qld-gov-au/quickstart-atlassian-bitbucket/blob/d6ebe59b5ccdd204a7edc72ab6f0f89d575ac6f8/templates/quickstart-bitbucket-dc.template.yaml 



(non-gist version below)

#Network Load Balancer health checks, need internal ip to approve connectivity

InternalNLBIp4List:
  DependsOn: NetworkLoadBalancerELB2
  Type: Custom::InternalNLBIp4ListCollector
  Version: 1.0
  Properties:
    ServiceToken: !GetAtt InternalNLBIp4ListCollector.Arn
    ELBv2Arn: !Ref NetworkLoadBalancerELB2
    StackName: !Ref 'AWS::StackName'
InternalNLBIp4ListCollector:
  Type: "AWS::Lambda::Function"
Properties:
    Handler: index.lambda_handler
    Role: !GetAtt InternalNLBIp4ListCollectorExecutionRole.Arn
    Runtime: python3.7
    Timeout: 120
    Code:
      ZipFile: |
        import cfnresponse
        import boto3

        def lambda_handler(event, context):
          elbv2 = boto3.client('elbv2')
          ec2 = boto3.client('ec2')
          elb2arn = event['ResourceProperties']['ELBv2Arn']
          response = elbv2.describe_load_balancers(LoadBalancerArns=[elb2arn])
          name = response['LoadBalancers'][0]['LoadBalancerName']
          elbtype = response['LoadBalancers'][0]['Type']
          filters = [{'Name': 'description', 'Values': ['ELB '+ elbtype[0:3] + '/' + name + '*']}]
          eni_response = ec2.describe_network_interfaces(Filters=filters)
          ip_addresses = [eni['PrivateIpAddress'] for eni in eni_response['NetworkInterfaces']]
          ip_addresses_cidr = [eni['PrivateIpAddress'] + '/32' for eni in eni_response['NetworkInterfaces']]
          print (ip_addresses)
          responseData = {}
          responseData['PrivateIpAddresses'] = ip_addresses
          responseData['PrivateIpCidrAddresses'] = ip_addresses_cidr
          cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData)

InternalNLBIp4ListCollectorExecutionRole:
  Type: "AWS::IAM::Role"
Properties:
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - sts:AssumeRole
    Path: "/"
Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
Statement:
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: !Sub "arn:${AWS::Partition}:logs:*:${AWS::AccountId}:log-group:/aws/lambda/*InternalNLBIp4ListCollector*"
- Effect: Allow
              Action:
                - "elasticloadbalancing:DescribeLoadBalancers"
- "ec2:DescribeNetworkInterfaces"
Resource: "*"
#NLB ip's need to be whitelisted to allow health checks to pass
SecurityGroupIngressNLB:
  DependsOn:
    - InternalNLBIp4List
    - SecurityGroup
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref SecurityGroup
    IpProtocol: "-1"
FromPort: -1
    ToPort: -1
    CidrIp: !Select [ 0, !GetAtt InternalNLBIp4List.PrivateIpCidrAddresses ]

SecurityGroupIngressNLB2: #ELB in 2 subnets, will have 2 ip's
DependsOn:
    - InternalNLBIp4List
    - SecurityGroup
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref SecurityGroup
    IpProtocol: "-1"
FromPort: -1
    ToPort: -1
    CidrIp: !Select [ 1, !GetAtt InternalNLBIp4List.PrivateIpCidrAddresses ]

Tuesday, April 20, 2021

Major change between spring-boot 2.2.x to 2.4.x with spring-cloud and spring-cloud-starter-aws-parameter-store-config (bootstrap.yml, profiles and more)

So with the change in how versioning works. A change to how config is loaded by default and a split between cloud agnostic spring-cloud and vendor specific integration types. There has been a major shake up on how to upgrade to the next minor version of spring-boot.


<parent>

        <groupId>org.springframework.boot</groupId>

        <artifactId>spring-boot-starter-parent</artifactId>

-        <version>2.2.11.RELEASE</version>

+        <version>2.4.4</version>

        <relativePath></relativePath><!--empty to not look up parent folder which is a helper pom on building-->

</parent>


-<spring-cloud.version>Hoxton.SR5</spring-cloud.version> <!-- https://spring.io/projects/spring-cloud release trains, Greenwich 2.1.x, Haxton 2.2.x -->

+<spring-cloud.version>2020.0.2</spring-cloud.version> <!-- https://spring.io/projects/spring-cloud release trains, Greenwich 2.1.x, Haxton 2.2.x, 2020.0.2 2.4.x -->


We now need to include io.awspring.cloud:spring-cloud-aws-dependencies as its now not included in the upstream org.springframework.cloud:spring-cloud-dependencies

    <dependencyManagement>

        <dependencies>

            <!-- spring cloud and aws cloud for param store lookup -->

            <dependency><scope>import</scope><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-dependencies</artifactId><version>${spring-cloud.version}</version><type>pom</type></dependency>

+            <dependency><scope>import</scope><groupId>io.awspring.cloud</groupId><artifactId>spring-cloud-aws-dependencies</artifactId><version>2.3.1</version><type>pom</type></dependency>

..

          

          

With dependencies we don't include spring-cloud-starter any more but with bootstrap.yml not being the 'default' way for loading we now need to include spring-cloud-starter-bootstrap to re-enable that functionality

-        <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter</artifactId></dependency>

-        <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>


+        <!-- spring-cloud-starter-bootstrap required to enable bootstrap.yml due to it not being default anymore -->

+        <dependency><groupId>org.springframework.cloud</groupId><artifactId>spring-cloud-starter-bootstrap</artifactId></dependency>

+        <!-- aws parm store config changed home-->

+        <dependency><groupId>io.awspring.cloud</groupId><artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>

          

          

You now can't have spring profiles load other profiles, you can do profile groups, but that is limited if you wanted something dynamic like enabling proxy settings. (this is maven profile adding a spring profile)

<profile>

        <id>local-proxy</id>

        <activation>

            <property>

                <name>env.http_proxy</name>

            </property>

        </activation>

        <properties>

-                <springBootRunArguments>--spring.profiles.include=PROXY,</springBootRunArguments>

+                <springBootRunArguments>--spring.config.import=classpath:application-PROXY.yml</springBootRunArguments>

        </properties>

        <build>

            <plugins>

                <plugin>

                    <artifactId>maven-surefire-plugin</artifactId>

                    <version>${maven-surefire-plugin.version}</version>

                    <configuration>

                        <systemPropertyVariables>

-                            <spring.profiles.include>PROXY</spring.profiles.include>

+                            <spring.config.import>classpath:application-PROXY.yml</spring.config.import>

                        </systemPropertyVariables>

                    </configuration>

                </plugin>

                <plugin>

                    <artifactId>maven-failsafe-plugin</artifactId>

                    <configuration>

                        <systemPropertyVariables>

-                            <spring.profiles.include>PROXY</spring.profiles.include>

+                            <spring.config.import>classpath:application-PROXY.yml</spring.config.import>

                        </systemPropertyVariables>

                    </configuration>

                </plugin>

                <plugin>

                    <groupId>org.springframework.boot</groupId>

                    <artifactId>spring-boot-maven-plugin</artifactId>

                    <configuration>

-                        <jvmArguments>-Dspring-boot.run.jvmArguments='-Dspring.profiles.include="PROXY"'</jvmArguments>

+                        <jvmArguments>-Dspring-boot.run.jvmArguments='-Dspring.config.import="classpath:application-PROXY.yml"'</jvmArguments>

                    </configuration>

                </plugin>


            </plugins>

        </build>

    </profile>

          

Any properties files you may have loaded that used "spring.profiles.include" can't be used any more with 2.4.x+ (unless you enabled legacy which will be going away after 2.6.x? version i believe)

          

-spring.profiles.include: defaults

+spring.config.import: classpath:application-defaults.yml

          

Also ensure that your bootstrap.yml has

aws.paramstore.enabled: true

It is on by default (But as its nowvin your config you can set it to false for local run's)

          

For more info, see:

https://spring.io/blog/2020/08/14/config-file-processing-in-spring-boot-2-4

https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#boot-features-external-config-files-profile-specific

https://github.com/awspring/spring-cloud-aws/blob/2.3.x/docs/src/main/asciidoc/parameter-store.adoc

https://github.com/awspring/spring-cloud-aws/blob/2.3.x/spring-cloud-starter-aws-parameter-store-config/src/test/java/io/awspring/cloud/autoconfigure/paramstore/AwsParamStoreBootstrapConfigurationTest.java

https://stackoverflow.com/questions/64907675/including-profiles-in-spring-boot-2-4-0-version

https://stackoverflow.com/questions/64994034/bootstrap-yml-configuration-not-processed-anymore-with-spring-cloud-2020-0

https://stackoverflow.com/questions/65063402/why-bootstrap-properties-is-ignored-by-spring-cloud-starter-config

https://docs.awspring.io/spring-cloud-aws/docs/2.3.0/reference/html/index.html#integrating-your-spring-cloud-application-with-the-aws-parameter-store

          

          

Something I want to look into is setting up something like https://github.com/localstack/localstack within a maven project so that param store loading can be tested in pdev instead of being caught in an aws dev/test environment.

Wednesday, October 07, 2020

Spring cloud param store Hoxton.SR6 to Hoxton.SR8 how to run locally

 

So your using aws param store to configure your application when deployed to docker/elastic beanstalk but ran into issues on doing testing after updates to remove cve issues.

before hand

at or before Hoxton.SR6 you only needed in test/resources/bootstrap.yml

aws.paramstore.enabled: false

But after updating to 
<spring-cloud.version>Hoxton.SR8</spring-cloud.version>
<dependency><scope>import</scope><groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version><type>pom</type></dependency>
<dependency><groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter</artifactId></dependency>
<dependency><groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId></dependency>

it now throws logs of errors and fails to boot, this sucks :'(
errors are:

see Log file output
lets work out where it broke. We first need to see what jar's were imported by our spring-cloud-dependencies

we do this by looking at 

https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dependencies/Hoxton.SR6 to https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-dependencies/Hoxton.SR8

And we notice that that the aws cloud version incremented from 2.2.2 to 2.2.4

to see what changed we can do this on github by visiting the link below

https://github.com/spring-cloud/spring-cloud-aws/compare/v2.2.2.RELEASE...v2.2.4.RELEASE

What was added in the doc's which looked like it might be our problem.

|aws.paramstore.region | | If region value is not null or empty it will be used in creation of AWSSimpleSystemsManagement.

|aws.secretsmanager.region | | If region value is not null or empty it will be used in creation of AWSSecretsManager.

On application startup, for its internal purposes Spring Cloud AWS performs a check if application runs in AWS cloud environment

by using `EC2MetadataUtils` class provided by AWS SDK. Starting from version 1.11.678, AWS SDK logs a warning message with exception when this check is made outside of AWS environment.

This warning message can be hidden by setting `ERROR` logging level on `com.amazonaws.util.EC2MetadataUtils` class.

so it seems we now need to set a region to block auto region lookup even when we have the paramstore disabled, we also need to do it for the stack and region lookup outside of paramstore.

test/resources/bootstrap.yml

aws:
paramstore:
enabled: false
fail-fast: false
region: "ap-southeast-2"
secretmanager:
region: "ap-southeast-2"

cloud:
aws:
region:
auto: false
static: "ap-southeast-2"
stack:
auto: false
Now that we did this, we are now not crashing. Awesome :D

Hope this helps others including future me.

Wednesday, July 17, 2019

AWS: Why does Auto Minor Version Upgrade Flag not upgrade to latest minor Version of Database.

Shout out to Asmita V. for this answer.

------------------------------------------
PROBLEM DESCRIPTION
------------------------------------------

Why were the RDS PostgreSQL instances not automatically upgraded to the latest Minor Version available even though the Auto Minor Version Upgrade Flag is enabled for your instances.

------------------------------------------
RESPONSE
------------------------------------------

The reason why your instances was not upgraded to these minor version is that AMVU will only upgrade the engine version for your RDS instance if the current engine version is being deprecated or the new one contains very important cumulative bug fixes and an upgrade is absolutely necessary. 

Please note that while we highly recommend that you perform an upgrade to 10.9, this upgrade will not happen automatically as of now using AMVU as the automatic upgrades happen only when absolutely necessary and you can also view such actions using describe-pending-maintenance-actions command.

If there is an auto minor version upgrade scheduled as a maintenance, please be assured that you will get a separate notification explicitly mentioning the same. Currently, in this case the upgrade will have to be applied manually.

Further, at your end, you can check if the minor version upgrade will happen automatically or not by using the below CLI command:

$aws rds describe-db-engine-versions --output=table --engine postgres --engine-version 10.6

Output:
||+-------------------------------------------------------------------------------------------+||
|||                                    ValidUpgradeTarget                                     |||
||+-------------+---------------------+-----------+----------------+--------------------------+||
||| AutoUpgrade |     Description     |  Engine   | EngineVersion  |  IsMajorVersionUpgrade   |||
||+-------------+---------------------+-----------+----------------+--------------------------+||
|||  False      |  PostgreSQL 10.7-R1 |  postgres |  10.7          |  False                   |||
|||  False      |  PostgreSQL 10.9-R1 |  postgres |  10.9          |  False                   |||
|||  False      |  PostgreSQL 11.1-R1 |  postgres |  11.1          |  True                    |||
|||  False      |  PostgreSQL 11.2-R1 |  postgres |  11.2          |  True                    |||
|||  False      |  PostgreSQL 11.4-R1 |  postgres |  11.4          |  True                    |||
||+-------------+---------------------+-----------+----------------+--------------------------+||

As you can see from the above output, for 10.6 version "AutoUpgrade" column is marked as "False" for all minor version upgrade (either 10.7 or 10.9) . So, it has to be done manually. Please make sure to upgrade to the latest minor version  (10.9)  so that you wont be prone to any security vulnerabilities as per the following notice:

[+] https://www.postgresql.org/about/news/1949/

AWS: How should we upgrade PostgreSQL 10.6 to 10.9 version of CloudFormation-controlled RDS instances

Shout out to Asmita V. for this answer.

Upgrade RDS PostgreSQL instances from 10.6 to 10.9 and in this process you want to understand that whether setting the "AllowMajorVersionUpgrades" flag in the CloudFormation template be sufficient and if during the process, existing instances will get replaced.

------------------------------------------
RESPONSE
------------------------------------------

Upgrading the PostgreSQL instance from version 10.6 to 10.9 is a minor version upgrade and hence would not require you to change the value of the parameter "AllowMajorVersionUpgrades" in your cloudformation template.

In order to upgrade your instances from 10.6 to 10.9 you can make a modification in your CFN stack by just specifying the "Engine Version" as 10.9 instead of 10.6 in your template. There is no replacement of the existing instance and hence there will be no loss of data. The existing instances will go to a "Modifying" State.

In order to confirm this behavior, I tried upgrading resources in my test environment and following are my observations:

---------------------------------------------
Testing
---------------------------------------------
Please refer the following set of steps that I took in order to upgrade my instance to 10.9 from 10.6:

Sample Stack
============================

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "PostgreSQL RDS Template ",
  "Resources": {
    "pgDB" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        "DBName" : "mydb",
        "DBInstanceClass" : "db.t2.small",
        "AllocatedStorage" : "20",
        "Engine" : "postgres",
        "EngineVersion": "10.6",
        "MasterUsername" : "username",
        "MasterUserPassword" : "password",
        "AutoMinorVersionUpgrade": false
      }
    }
  }
}

Step 1: Make Changes to the existing template
=======================================

1. On the Stacks page of the AWS CloudFormation console, click the name of the stack that you want to update.  https://console.aws.amazon.com/cloudformation
2. Select the Template tab and select View on Designer.
3. Modify the CFN Stack : "EngineVersion": "10.9"
        {
          "AWSTemplateFormatVersion" : "2010-09-09",
          "Description" : "PostgreSQL RDS Template ",
          "Resources": {
            "pgDB" : {
              "Type" : "AWS::RDS::DBInstance",
              "Properties" : {
                "DBName" : "mydb",
                "DBInstanceClass" : "db.t2.small",
                "AllocatedStorage" : "20",
                "Engine" : "postgres",
                "EngineVersion": "10.9",
                "MasterUsername" : "username",
                "MasterUserPassword" : "password",
                "AutoMinorVersionUpgrade": false
              }
            }
          }
        }
4. Validate you template.
5. Select Save. Save your template to S3 bucket.
6. Click on Save and copy the URL.
       
Ex. https://s3-external-1.amazonaws.com/cf-templates-us-east-1/template1
       
Step 2: Create Change Set for Current Stack
=======================================

1. On the Stacks page of the AWS CloudFormation console, click the name of the stack that you want to update.  https://console.aws.amazon.com/cloudformation
2. Go to Stack Actions and select "Create change set for current stack".
3. Select "Replace current template"
4. Input the URL that was copied in the Step 1:6.
5. On the Review page, click on Create Change Set.
6. In the preview page, under the Changes you will notice "Modify" under the Action column.
7. Click on Execute.

You can now check on the RDS Console. The status of the RDS instance would have gone to "Upgrading".

However, please note that engine version upgrade (major or minor) is always associated with some amount of downtime.

Even if your DB instance is in a Multi-AZ deployment, both the primary DB instance and standby DB instances are upgraded. The writer and standby DB instances are upgraded at the same time, and you experience an outage until the upgrade is complete.

Therefore it is always recommended to plan your upgrades in the non business hours.

[+] Upgrading the PostgreSQL DB Engine for Amazon RDS - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html

[+] Modifying a DB Instance Running the PostgreSQL Database Engine  - Settings for PostgreSQL DB Instances - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ModifyPostgreSQLInstance.html#USER_ModifyInstance.Postgres.Settings

---------------------------------------------
Conclusion
---------------------------------------------
Therefore, as per my testing, I can confirm that there will be no replacement of the existing instance during the process of upgrading PostgreSQL instance from  v10.6 to 10.9.
You may also go ahead and follow the steps given above in order to upgrade your instances.

---------------------------------------------
REFERENCES
---------------------------------------------

[+] Updating Stacks Using Change Sets  - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html

[+] Creating a Change Set  - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets-create.html