Lambda and EFS

AWS Lambda and EFS Troubleshooting

Being able to connect your AWS Lambda Serverless function to an EFS file system opens the door to a lot of new uses for Lambda functions.  You are no longer limited to 512 MB of /tmp/ storage, which is shared across all active invocations.  You can use the EFS file system to read, write, and work with large files, or large sets of files easily.  Unfortunately there are a few common issues and errors you may encounter while linking your EFS file system to your Lambda function.

“The function couldn’t connect to the Amazon EFS file system with access point”

If you are having trouble connecting to your EFS File System Access Point from your AWS Lambda Function, and you get an error like “The function couldn’t connect to the Amazon EFS file system with access point ….”  the issue is likely a networking problem you need to solve with security groups.

By default your EFS File System will take on the default Security Group for your VPC for each AZ connection.  If that Security Group doesn’t allow for NFS (TCP port 2049) traffic from the Lambda to the EFS file system, you will get the error above.  Personally I don’t like to use the default security group for much, as it can either be too permissive, or too automatic.  I want all my networking rules to be specific and applied with forethought.

To solve this issue, I recommend creating a new Security Group, under the VPC area of the AWS Console.  I named mine “VPC-EFS-Access”.     For my needs I am happy to allow NFS access to the EFS file system from anywhere in my VPC.  I setup both Inbound Rules and Outbound Rules (although I think the Outbound rules may be unnecessary).  On both Inbound and Outbound I allow traffic on TCP port 2049 (NFS) and for Source and Destination I add a rule each for both the VPC’s IPv4 subnet, and the IPv6 subnet.

EFS-SG-Inbound
EFS-SG-Inbound

 

 

 

 

 

 

 

 

 

 

 

Then go into the EFS section of the AWS Console, go into File Systems.  Then select the EFS File System you are trying to mount from your AWS Lambda function.  Click on the Network tab (far right), and then click on the Manage button (right side of the screen).  Now for each of the Availability Zones, select your new Security Group and add it to each AZ, and Save.

This should allow your Lambda to access your EFS File System without connectivity or network issues!

 

“EACCES: permission denied”

If you get an error accessing files (reading or writing) on your EFS mount from your AWS Lambda function, it may be due to a path mismatch.

This wasn’t clear to me while setting things up initially, but the EFS Access Point’s Root Directory Path setting has to align with the Lambda’s EFS Mount’s Local Mount Path setting.  Basically the EFS Access Point’s Root Directory Path has to be the same as the EFS Mount’s Local Mount Path, minus the “/mnt” prefix, and vice versa.

If you leave the EFS Access Point’s path to the default of “/”, then the Lambda’s Mount path must be “/mnt/”.  If you want the Lambda Mount path to be “/mnt/myproject” then you MUST set the EFS Access Point’s path to “/myproject”.  Once these settings align, you should be able to access the EFS file system from your AWS Lambda function.

 

If you run into any other issues mounting your EFS volume for your AWS Lambda functions, let me know!  I can add solutions to common issues to this post!


Posted

in

by

Comments

20 responses to “AWS Lambda and EFS Troubleshooting”

  1. Mano Avatar
    Mano

    In EFS Access point I have give as /opt/test/packs. In Lambda I have given /mnt/efs. When I am listing the files from /mnt/efs I can see all the folders inside that. but when I was trying to create a file inside the /mnt/efs i am getting the “errorMessage”: “[Errno 13] Permission denied: ‘/mnt/efs/demo/requirements-tests.txt’”,

    1. Devon Avatar

      Mano, try to fix your paths to line up as I’ve discussed in the last section of this article. If you leave the Access Point path as “/opt/test/packs”, then try the Lambda mount path of “/mnt/opt/test/packs”. See if that fixes your error.

  2. Shebin Avatar
    Shebin

    Hi Devon. thanks a lot for sharing this article. I was able to quickly fix this issue with your help.

  3. Jeanpierre Avatar
    Jeanpierre

    Lifesaver for mount points prefix issue.
    Thank you!

  4. Stan Bienaives Avatar
    Stan Bienaives

    Thanks a lot! This has been very helpfull.

    However notice that a lambda from a VPC looses access to the internet. To fix that you can follow this guidelines: https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/?nc1=h_ls

    1. Devon Avatar

      Very good point!

  5. Erick Avatar
    Erick

    Thank you!

  6. Will Avatar
    Will

    Life Saver

  7. rb Avatar
    rb

    Thank for the article. Very useful. I found that access point with / mounts to /mnt/zip worked fine.
    However, access point of /zip mounted to /mnt/zip did not work.

  8. Reza Avatar
    Reza

    This was very helpful, thanks !

  9. Steve Avatar
    Steve

    This doesn’t work for me. I followed the exact steps. On the security group of the lambda itself is where I believe the issue is. What exactly should the inbound rules be for the SG on the lambda itself if I am not allowed to allowed any IP in?

    1. Devon Avatar

      @Steve – the SG info above is only about allowing access from the Lambda to the EFS Mount. It sounds like your issue might be related to inbound traffic to the Lambda? I am using an API Gateway in front of my Lambda functions, so access can be controlled like this: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.html . Resource policies might be what you are looking for. If I’m misinterpreted your comment, let me know!

      If you do want to manage direct access to your lambdas via SGs (without using the API Gateway) then I think you’d just need to setup the normal inbound rules on that SG (80/443, etc…).

  10. Joe Avatar
    Joe

    Hi Devon, I have mounted and EFS on my ec2 FileSystem successfully. I created a new Lambda function that needs to access the same mounted access-point on ec2.
    I have mounted the same EFS, remote access-point with all SG and granted all needed permission. But for some reason the Lambda function cannot read a file from that Directory.
    On the other hand, I wrote a simple python script that list the directories on that EFS but it seems as if this Lambda is mounted to a different EFS.
    I see different folders. What should the Local mount path set if the remote access point is /tsl/midd-data for example.
    Can anyone assist please? I have been struggling with this issue for one week without any luck.

    1. Devon Avatar

      @Joe – Seeing different folders is suspicious. I assume not just different levels of the directory tree, but completely different files/folders? Without seeing your EFS config, EFS Access Point config, Lambda config, etc… its really hard to say what might be going on here….

  11. Satishchandra Singh Avatar
    Satishchandra Singh

    Hi Devon, thank you so much for putting up this article. It was very helpful. Saved my day. Don’t know why software documentations can’t be specific about things. God bless you.

  12. Joseph Avatar
    Joseph

    Hi Devon, Where to find source IP address when updating SG to allow traffic on TCP port 2049 (NFS)?

    1. Devon Avatar

      In my case I am allowing access from ANY IP in my VPC subnet. So the source subnets are my VPC subnets. And I make sure my Lambdas are running with in my VPC.

  13. Hakuro Tuqivrima Avatar
    Hakuro Tuqivrima

    Thank you for this helpful tutorial!!!!!!!

  14. Vinay Babu Avatar
    Vinay Babu

    Im using terrafrom to create efs and lambda, i have mouted efs on lambda , so when i update the efs, efs is gettng demoutned from lambda and we are receiving the following error.

    Error : Calling the invoke API action failed with this message: The function couldn’t connect to the Amazon EFS file system with access point arn:aws:elasticfilesystem:us-east-1:XXXXXXXXXX:access-point/fsap-08dd1c97f53a5ee25. Check your network configuration and try again.

    1. Devon Avatar

      Vinay,

      I expect that is “expected” behavior as when you’re updating the EFS resource in your Terraform, it’s destroying and re-creating it, or at least having to unmount it. I don’t use Terraform much so I don’t know what the best practice here is, but perhaps you could do something like create a new version of the lambdas, using the new EFS, and repoint the alias when that’s done, before doing any destructive work on the old EFS, although would probably depend on your EFS change needs, etc…

Leave a Reply

Your email address will not be published. Required fields are marked *

PHP Code Snippets Powered By : XYZScripts.com