Skip to content

[QUESTION] Mitigation around AWS RDS write forwarding in read only replicas #293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 tasks done
ls-urs-keller opened this issue Nov 20, 2024 · 6 comments
Open
3 tasks done
Assignees
Labels
need-more-info requested for more information question Further information is requested

Comments

@ls-urs-keller
Copy link

Summary

We had some issues around AWS RDS with cross region replication and rw/ro replicas with write forwarding.
I believe it is a bug on the AWS side. But it would be good if it could be mitigated.

Analysis here: https://github.com/ls-urs-keller/r2dbc-test

Mitigation: don't issue "START TRANSACTION READ WRITE" but either just "BEGIN" or "START TRANSACTION" if the transaction is readonly=false.

Checklist

  • I have searched the existing issues to make sure my question has not been asked before
  • I have provided all the necessary information and context to help answer my question
  • I have read the documentation and/or README for this project (if applicable)
@ls-urs-keller ls-urs-keller added the question Further information is requested label Nov 20, 2024
@ls-urs-keller ls-urs-keller changed the title [QUESTION] Your question title here [QUESTION] Mitigation around AWS RDS write forwarding in read only replicas Nov 20, 2024
@jchrys
Copy link
Collaborator

jchrys commented Nov 21, 2024

Thanks for providing the details that really helps. Let me check if we can implement this without affecting current behavior.

@jchrys jchrys self-assigned this Nov 21, 2024
@jchrys jchrys added the enhancement New feature or request label Nov 21, 2024
@ls-urs-keller
Copy link
Author

Thanks for providing the details that really helps. Let me check if we can implement this without affecting current behavior.

I have a support call with AWS, they can't reproduce, before changing anything let me come back to you with the results of that call.

@jchrys
Copy link
Collaborator

jchrys commented Nov 25, 2024

Got it :D

@ls-urs-keller
Copy link
Author

ls-urs-keller commented Feb 19, 2025

Got it :D

Sorry for the late reply.
So AWS support says it is a permission problem on our side. To use START TRANSACTION READ WRITE you will need more permissions than when using BEGIN or START TRANSACTION go figure.

I guess to make this work with AWD RDS and write forwarding you must not not issue START TRANSACTION READ WRITE.

@jchrys
Copy link
Collaborator

jchrys commented Feb 19, 2025

Thanks for update :D
I’ll check if we can adjust our implementation accordingly.

@jchrys jchrys added this to the Next milestone Feb 19, 2025
@jchrys
Copy link
Collaborator

jchrys commented Apr 6, 2025

Hi, @ls-urs-keller .
I’ve tested MySQL 5.7 with Aurora 2.11.5 in a global database setup by creating a read cluster in a different region and enabling global write forwarding. However, I couldn’t reproduce the issue using the provided repository. My test involved running SET SESSION aurora_replica_read_consistency = 'GLOBAL'(EVENTUAL, SESSION), and the results were as expected.

Image Image Image

Since the queries in the reproducible repository do not include SET aurora_replica_read_consistency, how do you set your consistency level? Also, if you could share or transfer your AWS support ticket, that would be really helpful for further troubleshooting. Could you please also provide your environment details, including the outputs of SHOW VARIABLES LIKE '%aurora%' and SHOW VARIABLES LIKE '%read%'?

Thank you!

@jchrys jchrys added need-more-info requested for more information and removed enhancement New feature or request labels Apr 6, 2025
@jchrys jchrys removed this from the Next milestone Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need-more-info requested for more information question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants